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1. INTRODUCTION 


1.1 Summary 


This proposal is for a two-year program of research and development. 
The purpose of the program is to improve group decision making through the 
use of a computerized, interactive decision aiding system which essentially 
guides the decision making group through the problem at hand. The aiding | 
methodology combines natural-language elicitation of decision trees, on-line 
sensitivity analysis, real-time assessment of multi-attribute values for 
decision outcomes, and direct visual feedback on areas of intra-group 
conflict. Such aiding allows the group to focus constantly on the decision 
path of greatest potential payoff, and on the critical differences of | 
opinion along that path. As a result, the quality as well as the speed of 
group decision making is significantly enhanced. 


The specific objectives of the proposed program include the 
following: | | | 


(1) Develop computer programs for efficient, comprehensive, 
elicitation of decision trees from a decision making group. 


(2) Develop computer programs for identifying structural and 
numerical differences among the contributions of individual 
group members, for merging these contributions and for 

resolving the points of conflict. 


(3) Develop effective means for displaying to the group the 
results of the elicitation procedures and conflict analyses. 


(4) Integrate the various programs and techniques into a complete 


aiding system which can be readily transferred to other test 
environments. | 
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6) Experimentally test the group decision aid, using. a 
| variety of representative military decision problems, to 
demonstrate its advantages under realistic conditions of © 
use. | | | 


(6) On the basis of the development effort and the exper imental | 
| results, establish guidelines and recommendations for future 
military applications of the group decision aiding methodology. 


The proposed program differs from previous and present decision 
aiding work in several ways. For one, it deals explicitly with the decision 
making group, rather than the individual decision maker. Secondly, it 
features the use of interactive computer support in the problem formulation 
phase of decision analysis, rather than in the optimization phase, as is the 
usual case. Furthermore, emphasis is placed on methods for guidance and 
control of communication between the group members. Third, and most MPO rete 
it introduces new adaptive computer techniques, such as on-line sensitivity. | 
analysis; to the decision analysis process, and integrates these techniques 
with other computational approaches to form a complete decision aiding | 
system. | a | 


While the proposed development represents a new concept in aiding 
group decision making, scientifically and technically it rests ona solid 
_ base of previous accomplishment. Of particular relevance is the recent . 
successful use of multi-attribute utility feedback by Gardiner and Edwards 
to identify and resolve conflicts in group decision tasks, and the recent | | 
demonstration by Leal and Pearl of an interactive computer program for | 
conversational elicitation of complete decision structures. Perceptronics 
includes on its program team the principal investigators of both these | 7 
| projects, and has complete access to the methodology involved. This factor, — 
along with Perceptronics' broad capability in decision analysis, and in. 
computerized decision aiding, will help insure the success of the proposed 
program. | | | a ners 


1.2 Technical Approach 


1.2.1 Problem Statement. Constant escalation in weapons cost and 
- effectiveness, as well as the increasing complexity of international 
relations, makes military decision making more critical today than ever 
before. In today's military environment, most upper-level decisions are 
made by committees and staff groups. Typically, such groups contain experts 
from several speciality areas, who bring to the decision environment 
disparate sets of values. Decision time is usually limited, the decision 
making procedure is relatively unstructured, and intra-group conflicts 
arise on a broad variety of issues. Consequently the group is unable to 
consider the maximum set of alternatives, conflicts are not resolved in 
an optimum manner, and the resultant decision is rarely up to the aggregate 
potential of the group membership. | | 


Decision analysis offers a promising approach to solving these 
problems. The analytical procedure of building a decision tree formalizes 
the decision process, and permits incorporation of individual values 
(utilities) into the selection of alternative courses of action (Hays, 
O'Connor, Peterson, 1975). However, decision analysis as it is usually 
practiced isa highly personal and time-consuming process. Decision analysts 
are often called upon to assist in the solution of problems ranging over 
a large variety of domains. In most cases the decision analysts know far 
less about the problem-domain than do their clients. Thus their 
contributions are confined primarily to the phases of formalization and 
optimization. While optimization is usually computer assisted, the 
formalization phase invariably has been accomplished manually, using lengthy 
interviews of persons more familar with the problem area. This approach is 
generally incompatible with the conditions of command group decision making. 


Accordingly, it would be highly worthwhile to automate the formalization 
phase, using an interactive computer system to interrogate the group members 
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and to construct a decision tree based on their responses. Leal and Pear] 
(1976) have shown that automated tree elicitation from individuals is | 
feasible, and that on-line sensitivity analysis can be used to concentrate 
tree development on the branches with highest pay- -off, thus streamlining 

the entire decision analysis process. Likewise, Gardiner and Edwards (1975) | 
and Sheridan (1975) have shown that direct, real-time feedback of responses 
in group decision making focuses the effort on areas of real difference, while 
maintaining the advantages of full group participation. The supporting 
concepts and research evidence for an automated group decision aid are in 
existence. The present proposal describes how such an aid will be developed 
and tested, and reviews some of the major anticipated benefits. 


1.2.2 Decision Aiding ae proposed group decision aiding © 
system incorporates four main contributions. These are: | 


() Interactive elicitation of decision trees, including on- paane 
ponent nyt analysis. | 


(2) Multi-attribute aie of group utility-values at critical 
points in the tree. | 


(3) Computer support system, including continuous feedback of | 
group decision responses by means of large-screen, color 


graphics display. 


(4) Use of "intermediator" personnel to optimize interaction | 
of the group with the aiding programs. | 


These contributions are summarized briefly below. . 


| Interactive Tree Elicitation. Perceptronics plans to adapt for 
group use Dr. Antonio Leal's existing computer program for elicitation of 


decision structures (Leal, 1976; Leal and Pearl, 1976). This program, which 
is described in Section 2.3, uses an English-like conversational mode to build 
a decision tree by interrogating the decision maker regarding his decision 
alternatives, and the associated probabilities and utilities. Sensitivity 
analysis, based on heuristic tree search, is used to identify the most 
sensitive areas of the tree during the building process. This allows time 
and attention to be concentrated on expanding only the critical Scenarios 
(sets of alternatives) within the large overall structure. 


Automated tree elicitation can be applied to group decision making 
using two main elicitation modes. | 


(1) Elicit complete trees individually from all group members. 
Merge the individual trees during group interaction. 
(2) Elicit a single group tree during group interaction. 


Technical approaches to each mode are discussed in Section 2.5. It it shown 
that they have in common many programming requirements, but that the first 
mode involves considerably more technical difficulties, associated primarily 
with merging trees having inherent structural differences. On the whole, it 
would be more cost-effective to implement the most practical mode first, 
evaluate it, and modify it or move on to the other on the basis of actual 


_ test data concerning group reaction and decision making performance. 


Accordingly, it was decided to implement the second mode -- single group 
trees -- during the first-year program, because this approach involves the 
most direct extension of existing software, the most powerful application of 
On-line sensitivity analysis, the most straightforward use of multi-attribute 
utility measurement, and consequently, the highest immediate chance of 
decision performance improvement. At the same time, it is planned to analyze 
further the technical details of tree merging, and to use the aiding program 
to elicit trees from individuals for comparison with the group product. 
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This analysis and comparison will provide insight ‘into the potential 
advantages and problems associated with the elicitation and menger of - 
~ complete individual trees. 


Multi-Attribute Utility Analysis. One can expect several types — 
of intra-group conflicts during the tree-building process. Conflicts 
| regarding decision alternatives and their. possible outcomes are easily’: 
resolved by merger or trimming. Conflicts regarding probabilities can | 
also be handled in standard fashion. Conflicts regarding utilities, however, 
are anticipated to produce the most severe disagreements during group 
interaction, because they directly reflect differences in the value 
structures that group members bring to the decision problem. In such 
cases, multi-attribute utility analysis provides a means. for arriving at 
the required single utility by decomposing the ere alternative or | 
outcome into its constituent attributes or dimensions. | 


Gardiner and Ford (1976), as well as Sheridan and Sichenman (1976), 
have shown that through group elucidation of values on each attribute - 
separately, a more accurate picture of the utility conflict is achieved, and 
therefore agreement can be more readily reached. The process allows each 

_ decision maker to present his own viewpoint on the critical aspects of the ~ 
problem, while leading the group as a whole to an eventual. consensus. (Such 
a local decomposition is, of course, not necessary if everyone agrees 
immediately on global utility assignments for the set of alternatives. ) | 
Section 2.4 describes multi-attribute utility analysis as it applies to the 
proposed group decision aid. A good part of the development effort will 

be devoted to selecting the best way of representing to the group members _ 
the dimensions of conflict and the means recommended for resolution. _ 


Support System. The group-aiding computer programs will be 
developed on a DEC PDP/11 minicomputer under the UNIX operating system. 
This will insure maximum transferability of the programs among ARPA 
contractors and other potential military users. A version of the LISP | 
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programming language has been located which runs under UNIX. The tree 
elicitation program is presently written in LISP, and availability of this 
language will simplify its adaptation to the present application. Members 
of the decision making group will interact with the program through alpha-_ 
numeric terminals. These units will be selected to permit quick entry 

and verification of numerical as well as written data. Feedback of decision 
information to the group as a whole will be by means of large-screen 

color display. It is planned to configure an economical and effective 
display by driving an Advent Videobeam color television projector with 
the newly available DEC Model VT30 color video display controller. The VT30 
controller attaches directly to the PDP/11 unibus, and with its associated 
software, permits simple creation of alpha-numeric and graphical materials | 
in eight colors. The resulting displays are ideal for the proposed aiding 
system. | | | 


The Intermediator. A procedural, rather than technical, innovation 
is the proposed use of a trained "intermediator” to mediate between the 
aiding program and the actual decision makers. The intermediator is 
not a decision analyst, but is highly familiar with all program operations, 
and is consequently able to dispense with some of the lengthier man-computer 
dialogue necessary to elicit data from completely naive individuals. 

Sheridan (1975) has stressed the importance of the skilled moderator in 
successful applications of his Electronic Voting and Discussion Technique 
_(EVDT). Preliminary discussions by Perceptronics with command-level military 
personnel revealed a quick acceptance of the intermediator concept as a 

means of overcoming resistance to computer aiding, and inefficiencies in 

the aiding process, which come from unfamiliarity with the interactive 
procedures themselves. If properly employed, the intermediator should 

help speed decision making while imposing no outside pressure on the 

decision making group. | | 


1.2.3 Empirical Evaluation. It is planned to evaluate the decision aiding 
methodology, as it is developed, through observation of its use by 
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representative decision-making groups in solving actual decision problems. 
- The system factors which can be varied in such evaluation include: 


(1) Tree Elicitation Mode — 
(2) Intermediator's Role 

(3) Interactive Configuration 
(4) Presentation Format 

(5) Group Composition 

(6) Decision Problem Area | 


Of these, the last, "decision problem area", is of particular interest. The 
decision problem area selected should: (a) be credible and of actual interest 
to the military as well as to the specific group; (b) involve options that | 
are already familiar to or easily explainable to the selected subject 
population; (c) reasonably complex, to allow a good number of alternative 
actions and move-response sequences, but not beyond several] hours solution 
time; (d) have a value structure which includes significant judgmental 
elements. - 


- The area selected for the initial year's effort is that of counter- 
- terrorist activity. It is Felt that this subject is current, important, and 
meets all of the above criteria. A typical problem set-up, for example, | 
would be that an airliner has been commandeered by terrorists, or a facility 
has been occupied, and the command group has convened to decide how to deal 
with the problem. Intelligence on the terrorists and their geographical 
location is available. The starting options of negotiation, force, waiting, 
etc., spread out rapidly to a rich decision tree, which includes a large a 
number of highly charged outcomes -- death of hostages, political loss of 
face, etc. -- which require multi-attribute utility analysis. Use of such 
scenarios should provide a maximum amount of information on system and 

group effects. Other problem areas will be included for comparison | 
purposes in the second-year work. | | | 
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1.2.4 Effectiveness Analysis. As presented in Section 3.6, analysis of 
available evidence indicates that the proposed aiding technology will lead 
to a significant improvement in group decision making performance. 
Improvement will come from two sources: | | 


(1) Decision Quality 
(2) Decision Time. 


These are treated separately in the following paragraphs. 


- Decision Quality. Improvement in decision quality is expected to 
arise from (1) increased participation of group members in the decision 
making process; and (2) the effects of multi-attribute decision modeling. © 
A number of related studies have shown that better decisions follow fuller 
and more even participation of group members, and that direct feedback of 
decision responses facilitates group interaction. The proposed group 
tree elicitation procedure insures full participation of the group, while 
the proposed immediate response feedback through a large-screen graphics 
display provides the means for presentation and resolution of group | 
conflicts. More complete explication of group opinion, a better route 
to concensus, and less "noisy" decisions, also result from the use of 
multi-attribute modeling. And numerous studies, including several at 
Perceptronics, have shown that including a decision model in a closed- 
loop aiding system greatly improves the consistency and payoff of the 
resultant decisions. | | | 


Decision Time. Improvements in decision making time are expected © 
from (1) reduced ambiguity in problem formulation due to computer guidance 
of discussions, (2) primary attention to critical decision areas through 
sensitivity analysis, and (3) efficient resolution of group conflicts 
through multi-attribute decision analysis and response feedback. Active 
computer guidance is the key to decision aiding success. Automatic 
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specification of needed inputs minimizes the time spent floundering, defining _ 
terms, and clearing misunderstandings. Sensitivity analysis greatly reduces 
the area of the decision tree which must be investigated, thus reducing the 
associated discussion time. Finally, the time spent in resolving utility 
value disagreements is minimized by presenting cleanly the various attributes 
of the choice. | | | 


Improvement Ratio. Preliminary analysis of existing experimental — 
data indicates an improvement ratio of between 4:1 and 9:1 for decision _ | 
quality, and between 3:1 and 5:1 for decision time. Assuming a multiplicative 
relationship for the overall aiding system improvement ratio, we estimate | 

a range of improvement in group decision making performance between 12:1 and ~ 
45:1. Taking into account that there is some interdependence of quality | 

and time factors, we can use 25:1 as an initial estimate of overall performance 
improvement. | ae | : 


1.2.5 Program Plan. The two-year program calls for the following division 
of major program goals: | - : a Ee 


(1) First Year - Group aiding system initial implementation and 
evaluation. Final planning, detailed design, programming, 
equipment acquisition and installation of system to elicit © 
and analyze group decision trees. Initial system test using 
a number of three-man and four-man groups and several versions | 
of counter-terrorist decision problems. Comparison of group 
solutions with those elicited by computer from individuals. © 
Evaluation report. | | | 


(2) Second Year - Program expansion and modification as required, 
to include individual tree elicitation if desirable. Full- 
scale empirical investigation of major system variables, 
using additional problem areas for comparison. Final : 


preparation of software in transferable form. Formulation 
of guidelines for future development and application. | 


It is planned that the first-year effort itself will result in 
an aiding system that could be transferred to another suitable computer 
System, i.e., PDP/11 series with similar feedback display capabilities. 
The second-year development, however, will finalize the system, condensing 
the programs where possible and adding documentation directed specifically 
toward the applied user. | | 


1.3 Capabilities 


Perceptronics brings to this program a background of experience 
and expertise in a number of directly related R&D areas; these include: 


(1) Decision Aiding and Analysis 
(2) Adaptive Information Systems 
(3) Human Factors Design and Experimentation | 
(4) On-Line Software Development 


As a result, Perceptronics is able to provide this program with a 
particularly strong development team. The program will be directed by 
Dr. Amos Freedy, who has previously directed development of all adaptive | 
software at Perceptronics. Principal Investigator will be Dr. Antonio Leal, 
who developed the first computer program for interactive elicitation of 
decision trees while at UCLA, and who has been involved in the application 
of on-line, adaptive decision programs while at Perceptronics. Consultation 
in the area of decision and sensitivity analysis will be provided by 
Dr. Judea Pearl of UCLA, co-developer with Dr. Leal of the prototype tree 
elicitation program. Consultation in the area of multi-attribute utility 
measurement will be provided by Dr. Peter Gardiner of USC, who has worked 


directly in the group decision making area for the past several years. 
Human factors inputs to group display design will be coordinated by 

Dr. William Crooks, wh heads Perceptronics' engineering psychology group. 
Members of the support team will be selected from Perceptronics staff of 
system analysts, programmers, and engineers. Similar Perceptronics teams 
have performed and are now performing successfully on a number of related 
research and development programs for Navy, Army, Air Force, ARPA, NASA 
and industrial contractors. | 


Additional expertise in the area of decision analysis is provided | 
by Perceptronics' Decision Research Branch, in Eugene, Oregon. Decision 
Research Personnel include Dr. Paul Slovic, Dr. Sarah Lichtenstein and 
Dr. Baruch Fischhoff. They are among the foremost investigators in the 
psychology of judgment and decision making. Their broad background of 
experience provides a valuable source of theoretical and empirical 
information directly related to group decision aiding. 


Finally, the proposed program will be supported by a number of 
ongoing programs at Perceptronics. Included are: (1) Development of a | 
computer-based supervisory system for managing information flow in C3 systems, — 
being conducted for ARPA; (2) Development of adaptive decision aids for 
airborne ASW operations, being conducted for NADC and ONR; (3) Development 
of adaptive engagement and control logic for ballistic missile defense, 
being conducted for the Army's BMC under subcontract to TRW; (4) studies 
of man/computer communications in remotely controlled systems, conducted 
separately for ONR, and for NASA under subcontract to JPL; and (5) Study 
of requirements and methodology for decision training in operational systems 
being conducted for the Naval Training Equipment Center, Orlando. On these, 
and on all previous programs, the company has shown its ablility to adhere 
to firm schedules of achievement and cost. | 


2. TECHNICAL BACKGROUND 
2.1 Overview 


This chapter is an initial presentation of the several technical 
areas which will directly affect the planned development and evaluation of 
a computerized group decision aid. Beginning with a brief description 
of the underlying decision structure, it describes two existing methodologies 
relevant to computerized group decision aiding: elicitations of individual 
decision trees, and measurement of multi-attribute utilities in groups. 
Following this, means are discussed for extending automatic tree elicitation 
procedures to groups, and a method selected for initial implementation. 
Finally, a preliminary analysis is made of the benefits expected from the 
proposed aiding approach. | 


apd Structure of Group Decisions 


Group decisions are taken with regard to a large number of problems. 
However, we can consider that in many cases of military importance, the 
fundamental structure which models the decision making process is the 
decision tree. The basic building blocks of the decision tree are (1) 
decision nodes and (2) event nodes. Decision nodes represent a set of 
plausible, mutually exclusive alternatives. These have associated with 
them utility values that represent the worth of the particular alternative 
to the decision maker(s). Event nodes, on the other hand, represent 
possible consequences of the various alternatives the outcomes of which may 
not be under the direct control of the decision makers. Thus, in the process 
of tree building, events must be assigned estimates for the probability of 
their occurrence as well as utility values for their outcomes. 


The decision tree is in essence a structured relationship among 
decision nodes and event nodes, which defines possible future situations. 
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These situations (paths in fe tree) are iepaed "scenarios", since they 

- describe future real-world configurations that may or may not come to pass. 
It is precisely the exploration and comparison of many scenarios that allows 
the best currently available alternative to be chosen. The main objective 
of tree expansion is to determine which of the available scenarios is the 
best choice; that is, which has the highest expected utility value. In 
automated tree elicitation as proposed here, it is the purpose of sensitivity | 
analysis to insure that only the most fruitful and critical scenarios are 
explored to a sufficient level of detail for confident solutions, and the 
purpose of multi-attribute utility analysis to permit quick resolution of 
value conflicts within the exploration process. 


2.5 Computerized Elicitation of Decision Trees 


2.3.1 General. This section describes an interactive computer program 
for eliciting decision trees from decision makers, using an English-like 
conversational mode. The program was developed and implemented by Dr. 


Antonio Leal of Perceptronics as part of his Ph.D. program at UCLA, under the — 


supervision of Professor Judea Pearl. A full description of this program 
can be found in Dr. Leal's thesis (Leal, 1976); a shorter version will be 
published soon (Leal and Pearl, 1977). | | 


2.3.2 Rationale. Complex decision structures, or decision trees, are — 
generally elicited through personal interviews by decision analysts with 
the actual decision makers. From a practical viewpoint, the major drawback — 
of such interviews is their length and cost. Since real-time analysis of 
decision trees is beyond the limitation of human computational capability, 
it invariably happens that many hours of interviews are spent eliciting 
portions of the decision tree which do not have decisive bearing on the 
problem at hand. This fact can only be discovered at a later Stage, once 
the problem structure is formalized and a sensitivity analysis has been 
conducted on a computer. Therefore, if the computer could be brought into © 
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the elicitation process itself, it would not only eliminate the need for a 
highly trained analyst in attendance, but would also improve the effectiveness 
of the basic analysis procedure. 


2.3.3 Heuristic Search. The approach to automated elicitation described 
here centers on the realization that the process of conducting an elicitation 
dialogue is structurally identical to conducting a heuristic search on game 
trees, as is commonly practiced in Artificial Intelligence programs. Figure 
2-1 illustrates this analogy. Heuristic search techniques, when applied to 
tree elicitation, permit real-time rollback and sensitivity analysis as the 
tree is being formulated. Thus, it is possible to concentrate effort on 
expanding those parts of the tree which are crucial to the resolution of the 
solution plan. Heuristic search requires the decision maker to provide 
provisional utility values at each intermediate stage in the tree construction. 
These estimate the promise of future opportunities open to him from that 
stage. The provisional values serve a role identical to a heuristic 
evaluation function in selecting the next node (scenario) to be explored 

in more detail. 


What do such provisional values actually represent? They are no 
different than values placed at the tip nodes of the completed tree; i.e., 
estimates of the relative worth of the opportunity provided by each node. 
Since all utilities are merely estimates of the rollback value of a complete 
tree emanating from that particular point on, the difference between 
provisional utilities and final utilities is only their degree of accuracy. 
The former can be regarded as consisting of the latter plus a noise term due 
to deficiencies in mental aggregation procedures. Nevertheless, if the 
decision maker is requested to provide values for nodes as they are being 
expanded, the information can be used to determine a node expansion order. 
As the tree expands, these provisional values become (by rollback) more 
"refined", that is, closer to the "true", automatically processed, value. 
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FIGURE 2-1 
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2.3.4 Sensitivity Analysis. The actual order of node expansion is determined 
by sensitivity analysis. Since the only effect of expansion is to refine the 
provisional value assigned to the expanded node, it is reasonable to focus 

the expansion effort on those nodes which, if they suffer a change of value, 
would be most likely to influence the plan selection -- namely those most 
likely to impact the top contending alternative courses of action. This is. 
accomplished by ranking all the available tip nodes in order of sensitivity. 


The sensitivity measure is obtained by estimating the amount of 
change (differential) in a given provisional node value necessary to cause 
a change in the currently best initial decision. For example, in Figure 2-2, 
a partial tree is shown with initial decision branches b, > b,» and b,- 
Branch bs is shown with an expanded event node that has two outcomes A and B. 
Assume that from a previous rollback calculation, the values of the three 
decision branches are 5, 3, and 2 respectively. Thus, b, represents the 

currently most promising decision. : 


To calculate the sensitivity differential of node A, the following 
question is posed, "How much should the value of node A (currently 5) be 
raised so that the value of the initial branch leading to node A (i.e., b.) 
will equal the currently highest branch (i.e., b,)2" Branch bo must — 
obviously be incremented by at least 2, but node A is only contributing 20% 
of its value to it. Node A must be raised to a total higher than 15 in 
order to cause bs to exceed b,- Thus, the sensitivity differential of node 
A is 10. Similarly, B must be raised to 5 (assuming no other changes) in 
order to cause b, to become preferred to b,. Since A must be raised more 
than B, A may be said to be more "robust" than B. The general procedure 
for finding the sensitivity differential A(n) for any node n is given by 
the following recursive relation: 


A(n) 


P(n) for an event node n 
A(T(n)) = 


A(n)+V(n)-V(r(n)) for a decision node n 


where T(n) is a successor of node n and P(n) is the probability along the 
branch from n to r(n), and V(n) is the value of node n. 


2.3.5 Conversational Elicitation. The elicitation program as currently 
implemented constructs and expands a decision tree by gaining information 

— from a decision maker operating at an interactive computer terminal. 

Figure 2-3 shows the main parts of the procedure for expanding each node. The 
first step (1) is to choose a node for expansion on the basis of sensitivity 
analysis. The decision maker, after being alerted to direct his attention 

to a specific area of the tree, is interrogated about the characteristics of 
the node (2). The node is automatically classed as a decision or an event, 
and elicitation of decision alternatives or event outcomes, respectively, 

then begins (3). _ 


Using the decision maker's responses for feedback, the provisional 
values (4), probabilities (5), and costs (6) are requested as necessary. 
After a request (see below) for experimentation (7), the chosen node is said 
to be "expanded". Enough information has been obtained to permit rollback. 
The elicitation procedure is repeated when a new node is chosen. 


Experiment Node. To give the decision maker an opportunity to improve 
his probability estimates, a mechanism is provided in the elicitation program 
to represent the option of gathering information about uncertain events. 

This information gathering usually takes the form of an experiment with the 
following properties: | | 


1. SELECT NEXT NODE FOR EXPANSION 


| 


2. DETERMINE NODE TYPE 


3. ELICIT ALTERNATIVES OR OUTCOMES 


4. DETERMINE IF MUTUALLY EXCLUSIVE 


5. ELICIT PROVISIONAL VALUES 


6. ELICIT PROVISIONAL PROBABILITIES 


7. ELICIT COSTS 


8. REQUEST FOR EXPERIMENTATION 


cuctatiourmououe  PERCEPTRONICS 
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(1) The possible observations (experimental results) are known and 
are mutually exclusive. 


(2) The relations between the observations and the uncertain event 
are expressible in probabilistic terms. 


(3) If the experiment has a cost, it must be known. 


The "experiment" may represent either an actual physical act (search 
for data) or an internal mental act (recall pertinent information). The 
interactive program elicits the above information from the decision maker 
and incorporates it into the current tree by inserting an "experiment node" 
at the proper place. Then, the decision maker is requested to supply 
conditional probabilities relating each observation to each event outcome. 
This allows calculation (by Bayes rule) of the value of the experiment (if 
it involves a physical act) and of the refined (a posteriort) probability 

once the observations become known. | | 


Information gathering acts can result in excessive repetition, since 
the trees emanating from each experimental outcome are usually identical in 
structure. To eliminate difficulties connected with subtree duplication the 
entire experiment structure is represented by a single (diamond shaped) node 


with a single branch. Since the probability and value labels on the 


duplicated subtrees are not identical, but vary with the experimental outcome, 
these labels are represented in the form of vectors with one entry dedicated 
to each experimental outcome. This representation results in a "cleaner" 

tree structure, which more closely matches the user's perception of the | 
decision problem. | 


2.3.6 Example. Figure 2-4 reproduces directly the beginning of a conversation 


between the elicitation program and a decision maker. The decision under 
consideration is whether to send a project proposal to government agency Y 
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WHAT'S YOUR PROBLEM? 
*SENDING THE PROPOSAL TO AGENCY Y OR WAITING. 
PLEASE LIST THE ALTERNATIVES THAT YOU HAVE, ONE 


AT A TIME. | 

1. *SUBMIT TO AGENCY Y IMMEDIATELY 

2. *WAIT FOR TWO MONTHS 

3. *NO MORE 

ARE THE ALTERNATIVES MUTUALLY EXCLUSIVE? 

eYES 

~ NOW CONSIDER "SUBMIT TO AGENCY Y IMMEDIATELY”. 

ASSUMING THAT THE CURRENT SITUATION WERE TRUE, 
TRY TO ESTIMATE HOW GOOD IT IS INDEPENDENTLY OF 
OTHER POSSIBILITIES. YOU MAY USE MONEY AS A 
SCALE OR ONE OF YOUR OWN CHOOSING, HOWEVER, YOU 
MUST BE CONSISTENT AND USE THE SAME SCALE 
THROUGHOUT THE INTERVIEW. ae we 

DO YOU WISH TO USE MONEY AS A SCALE? 

SYES 

ENTER YOUR ESTIMATE. 

_ *50000 

ARE THERE ANY HIDDEN COSTS NOT TAKEN INTO ACCOUNT 
IN THE PREVIOUS VALUE ESTIMATION? 

*NO | 

LET'S LOOK AT "WAIT FOR TWO MONTHS" : 

WHAT VALUE WOULD YOU GIVE TO THIS See | 

*70000 

WHAT IMMEDIATE COST IS EXPECTED? 

*Q 


SUPPOSE THAT YOU HAD CHOSEN TO WAIT FOR TWO MONTH 
CAN YOU THINK OF THINGS THAT MAY HAPPEN AS A 


RESULT? 
*NO 
IS THERE A DECISION TO BE MADE AT THIS POINT? 
*YES 
STATE THE CHOICES THAT You HAVE. 
T. *SEND TO X 
2. *SEND TO Y 
3. *DONE | 
DOES THE CHOICE OF ONE ALTERNATIVE EXCLUDE THE 
OTHERS? 
*YES 


NOW CONSIDER "SEND To x" 


TRY TO PLACE A NUMERIC VALUE ON THIS ein? 


*70000 | 
WHAT WOULD BE THE IMMEDIATE COST, ASSUMING THIS 
a 


LET’ S LOOK AT "SEND TO Y" 
HOW WOULD YOU EVALUATE THIS SITUATION? 
*50000 
HOW MUCH EXPENSE IS ANTICIPATED? 
*0 


ASSUMING THAT "SUBMIT TO AGENCY Y IMMEDIATELY" was 


PICKED, ARE EVENTS ABOUT TO HAPPEN OVER WHICH 
YOU HAVE NO CONTROL? 

*NO 

DO YOU HAVE A CHOICE OF ALTERNATIVES? 

*NO | 

DO YOU WISH TO STOP EXPLORING FURTHER IN THIS 
DIRECTION? 

*OK 

SUPPOSE THAT "SEND TO Y" IS YOUR CHOICE, 
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ARE THERE SOME EVENTS THAT MAY HAPPEN? 
~ *YES 7 
ee LIST THE OUTCOMES. 
- *ACQUIRE FUNDING FROM Y 
2 *¥ REJECTS PROPOSAL 
3. *DONE 
ARE THESE EVENTS MUTUALLY EXCLUSIVE? 
*VES 
NOW CONSIDER "ACQUIRE FUNDING FROM Y" 
ESTIMATE THE VALUE IF YOU WERE IN THIS POSITION. 
*50000 
TRY TO ESTIMATE THE PROBABILITY THAT THIS EVENT 
WILL HAPPEN. 
*0.8 
IF THERE IS AN ASSOCIATED COST, WHAT IS IT? 


*0 

LET'S LOOK AT ny REJECTS PROPOSAL" 
WHAT IS THE SITUATION WORTH TO YOU? 
‘+ : | 


0 : 

STATE THE IMMEDIATE COST IF THERE IS ONE. 

*Q 

CAN YOU IMPROVE THESE PROBABILITY ASSIGNMENTS: 
BY PERFORMING AN EXPERIMENT? 

*NO- ? 

SUPPOSE THAT “ACQUIRE FUNDING FROM Y" HAPPENED. 

oc THERE SOME OPTIONS AVAILABLE TO YOU? - 

roa UNCONTROLLABLE OUTCOMES OCCUR? 


HAS THERE BEEN ENOUGH DETAIL EXPRESSED SO FAR? 
WHAT IF "Y REJECTS PROPOSAL" OCCURS? 
MOULD THERE BE OPPORTUNITIES OPEN TO YOU NOW? 
CAN YOU THINK OF THINGS THAT MAY HAPPEN AS A 
RESULT? 
+NO 
at WE TERMINATE THE INTERVIEW? 
WHAT IF YOU CHOOSE TO SEND TO xX? 
_ ARE EVENTS ABOUT TO HAPPEN OVER WHICH YOU HAVE NO 
CONTROL? 
#YES 
EXACTLY WHAT EVENTS COULD OCCUR? 
1. *FULL FUNDING FROM X 
2. *PARTIAL FUNDING FROM X 
3. *X REJECTS PROPOSAL 
4. *NO MORE a | 
DOES THE OCCURRENCE OF ONE EVENT EXCLUDE THE 
OTHERS FROM HAPPENING? 
#YES 
NOW CONSIDER "FULL FUNDING FROM X" — 


ae a a a ets Le aa teat Sat eta Salat SideeME SAK Ae abil 
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immediately for $50K, or in two months to agency X, for $100K. An asterisk 
is the signal from the computer that it is ready to receive input from the 
user. Thus, text immediately following an asterisk was typed by the 

decision maker. Figure 2-5 shows the full decision tree as elicited, and 
Figure 2-6 shows the condensed structure with experiment nodes inserted. The 
choice indicated is to wait for two months and submit to agency X, since the 
expected payoff from this action is $72.6K, against $50K from agency Y. | 


2.4 Multi-Attribute Utility Measurement 


2.4.1 Intra-Group Conflict. At some point in the discussion of utilities 
attributable to alternatives or outcomes, it is likely that members of the 
group will differ significantly on the values assigned. At this juncture 

a closer look at the utilities associates with that node must be made. By 
closer we mean shifting from "gestalt" or "holistic" value assignment to 
multi-attribute utility measurement (MAUM). The reason for this is that — 
arguments over the assignment of utility usually reflect genuine disagreements 
about values. Multi-attribute utility measurement, by breaking a holistic 
evaluation down into its component parts, can spell out explicitly what the 
values of each participant in the group are, show how and how much they 
differ, and in the process frequently reduce the expanse of such differences. 


Figure 2-7 illustrates a typical conflict situation. Three alternatives 
have been identified at a node. On two of these, the five group members are 
in good agreement with respect to overall value, and a mean value can be 
easily assigned. On the third alternative, the differences are much greater, 
and the conflict must be resolved in another way. | 
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Figure 2-7. VALUE ASSIGNMENT BY GROUP MEMBERS 


The approach is to pause in the assignment of value, shift to 
developing a group MAUM model, and use the group MAUM model to assign the 
overall value required. The overall value so obtained will emerge in one of 
several ways: (1) by group agreement fostered by use of the MAUM technique; 
(2) by a reselected combination rule; or (3) by the senior member of the 
group assigning the value (as any commander weighs the contributions of his 
staff to arrive at an overall group evaluation). At this point the program - 
receives an overall value for the alternative and the tree elicitation | 
process continues as before. 


2.4.2 Developing a Group MAUM Model. The basic idea of multi-attribute 
utility measurements is quite familiar (see, for example, Raifa, 1969). 

Every outcome of an action may have value on a number of different dimensions 
or attributes. The technique, in any of its numerous versions, is to discover 
those values, one dimension at a time, and then to aggregate them across 7 
dimensions using a suitable aggregation rule. Probably the most widely used 
type of aggregation consists of simply taking a weighted linear average. 
Only that procedure will be discussed here, since theory, simulation 
computations, and empirical experience all indicate that weighted linear 
averages yield satisfactory values, while remaining easiest to elicit and 
understand. The technique consists. of ten SEEPS: as follows (Gardiner and 
Ford, 1976): 


Step 1: Identify the group whose utilities are to be measured. Ina 
military setting this group is frequently the senior officer and his 
staff. Usually, several staff members have a stake in voicing a 
decision. | | 


Step 2: Identify the issue or issues to which the utilities needed 
are relevant. In this instance, the issue is formed by the "context" 
in which we are assigning utilities to the nodes of our decision tree. 
For example, an intelligence or logistics operation could be driving 

the decision process. 


Step 3: Identify the entities to be evaluated. The entities here 

are the branches of the example tree in Figure 2-7. Specifically, we 

focus on the branch where the large disagreement occurs about what 

utility to assign to the branch. The attempt is to try and figure out 

the reason for the utility differences (and if possible resolve them) 

by breaking down the holistic utility estimate into dimension-by- 
dimension estimates. | 


Step 4: Identify the relevant dimensions (attributes) of value the 
group considers in evaluating and assigning utility to the node 
branches. Here it would be practical and useful to establish a 
preliminary "menu" of military dimensions, and select those which 
apply in each particular instance. 


Step 5: Rank the dimensions in order of importance. This ranking |. 
task, like step 4, can be performed by each individual in the group 
separately, or by all members acting together as a group. We plan 

to use an individual process, and this is the first step where 
different values tend to emerge. For example, Bauer, Gebert and 
Meise (1973) found that individuals could agree on what constituted 
the goal fabric (i.e., importance dimensions). Disagreement in 


values began to emerge only when they were asked to rate dimensions | 
in importance and evaluate difference levels of performance on each. 


Step 6: Sum the importance weights, divide by the sum, and multiply 
by one hundred. This is a purely computational step which converts 
the importance weights to numbers that, mathematically, are rather 
like probabilities. The choice of one to one hundred scale is 
complete arbitrary. a | 


Step 7: Measure the utility of the branch on each dimension. In the 
usual application of MAUM, dimension-by-dimension utility curves are 
constructed for each group member by having each member draw a graph 
where the x-axis represents the range of performance for the entities 
being evaluated and the y-axis represents the utility associated with 
the corresponding x-axis performance. In this application, we | 
abbreviate the approach. The fundamental reason for having a utility 
curve on each importance dimension is to be able to determine the 
dimension-by-dimension contribution of utility for any number of 
decision alternatives. In this instance, where each time we use © 
MAUM we have only one entity to be evaluated (i.e., the branch where 
the holistic utility assignment cannot be agreed upon), we do not — 
need the whole utility curve for each dimension. We only need one 
point on each curve. The easiest way to obtain this single utility 
point for each dimension is simply to ask each group member for it 
directly. Each group member will be asked to consider the branch | 
_under discussion and the level of performance of this branch on each 
importance dimension identified. Then each group member, individually, 
assigns a utility number between zero and one hundred (the range of | 
the utility scale) dimension by dimension reflecting the utility of _ 
the branch's performance on each dimension. At the conclusion of 
this step we will have a "decomposed" version of Figure 2-7 such as 
shown in Figure 2-8. | | 


Bi Group Member | 
Importance ] 3 2 ae) 
Dimension Weight/ Weight/ Weight/ 


Utility Utility ... Utility 


1 2/50 7/90 .05/95 
2 .2/75 «2/05 05/05 

© 3 .3/20 — .05/50 .30/40 
4 3/65  —.05/50 .60/55 


Figure 2-8. DECOMPOSED UTILITY ESTIMATES 


For each importance dimension identified we have an importance weight 
and a utility estimate for how the branch performs on that dimension. 
And we have this for each group. | 


Step 8: Calculate the overall branch utility for each group member 
by using a simple weighted average: 
4 WwW 
Una = a21 Squlbg;) 
where Us is the utility of the branch for the ith group member , d is 
the number of importance dimensions, Wy is the importance weight for 


the dth importance dimension, and u(b,.) is the utility of the branch 
for the ith person on the dth importance dimension. 


Step 9: Assigning the branch a utility. The results of the | 

computation in Step 8 will produce one of three possible outcomes: 
(1) the individual utilities of the branch calculated according to 
the procedure of Step 8 will produce numbers so close for each member | 
of the group that further discussion is not needed and the group mean 
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is simply assigned to the branch and the process returns to the .__ 
previous mode of operation; (2) the individual utilities will differ 
and when plotted will produce a plot as shown in Figure 2-9; (3) the © 
individual utilities will differ and when plotted will produce a 
plot as shown in Figure 2-10. _ _ . 


2.4.3 Group Consensus. We expect that in some instances, Outcome 1 will 
actually happen. The mere process of going through the refined process of : 
assigning a utility to a branch will "foster" agreement. This phenomenon 
has been observed in previous MAUM applications. For example, Gardiner | 
(1974) compared the evaluation of 14 individuals, where the same list of 15 | 
| alternatives, each with 8 importance dimensions, were evaluated first by 
group Gestalt evaluations and secondly by group MAUM evaluations as proposed | 
here. Gardiner found and reported (Gardiner and Edwards, 1975) that the 

use of a group MAUM value model for evaluation instead of their intuitive 
Gestalt evaluation turned disagreement into substantial agreement. Ina 
typical case, groups were observed to differ by 100% on the total worth of 
an alternative when the worth was determined holistically. The difference 
was reduced to 17% when worth was determined by MAUM procedures.. 


Why? Here is a plausible answer. Making Gestalt evaluations, group 
members with strong points of view tend to concentrate on those aspects of — 
the entities being evaluated that most strongly engage their biases. The 
multi-attribute procedure does not permit this; it separates judgment of |. 
the importance of a dimension from judgment of where a particular outcome 
falls in that dimension. While different views may cause different thoughts 
about how good a procedural level of performance on some dimension may be, 
evaluation of other dimensions will be more or less independent of viewpoints. 
Agreement about those other dimensions tends to reduce the impact of ... | 
disagreement on a controversial dimension. That is, multi-attribute utility 
measurement procedures do not provide an opportunity for any one or two 
‘dimensions, or any one or two group members, to become so salient that they 
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FIGURE 2-10. A UTILITY DISAGREEMENT ON SPECIFIC DIMENSION(S) 


exaggerate existing sources of conflict and disagreement. While there may — 
well be differences in importance weights assigned and utilities, when the 
dimension-by-dimension utilities are aggregated, these differences are 
washed out and different MAUM models of different individuals end with 
assigning the same overall worth. 


2.4.4 Group Conflict. One cannot rely on characteristic MAUM to foster 7 
agreement, or even ma of cases. We guess that either the outcomes . 
shown in Figures or 2-4 will be more: ‘likely. In the instance of © \. 
Figure 2-9, the plot shows that two group members differ on the contribution 
of utility that the branch's performance on each dimension contributes to 
overall branch utility. But the difference is relatively constant and in 
the same direction. One possible explanation is an evaluation bias (e.g., 
the ith member just likes to assign higher value numbers than the jth member 
for any dimension). We suspect that if this kind of profile shows up, that 
a compromise can easily be reached, particularly if the constant evaluation 
bias explanation holds up, and that the compromise will involve some sort 

of "meeting each other halfway". 7 


In the instance shown in Figure 2-10 there is a “genuine” utility 
conflict with respect to importance dimension 2 with respect to branch 
performance. And this conflict (as well as other if present) can be examined | 
for their effect on overall utility assignment conflicts for the branch. If 
(as in the example shown in figure 2-4) most of the overall utility | 
disagreement can be attributed to a single importance dimension the debate | 
is focused for the group. Perhaps a short discussion can help clarify and ~ 

resolve the disagreement. Perhaps the disagreement cannot be amicably 
resolved. At this point any one of a number of strategies can be employed 
by the group leader to resolve the conflict (see Keeney» 1972). | 


2-20 


Feedback Display. Of particular importance is the group feedback 
display. As shown by Bauer, Gebert & Meise (1973), the dimension-by-dimension 
values should be displayed for the entire group. That is, the list of options 
to be assigned values should be displayed according to how they rate on each 
dimension, as seen by each member of the group. Sensitivity analysis can 

be performed to determine the exact importance dimensions (or single 
dimension) where the individual value assignments create the overall 
disagreement. In such instances, the one or two conflicting dimensions can 

be isolated and identified for further discussion. | 


Bauer and Vegener illustrate how such a display can be produced. 
With its aid, the senior member of a group (or the intermediator) can focus 
discussion on the crucial dimension. In the end, a group judgment can still | 
be reached either by vote (see Sheridgn and Sicherman, 1972) or by the 4 
senior member present announcing his decision based on the kind and magnitude 
of value conflict clearly outlined in the MAUM displays, and on the relative 
value he places on his individual staff members. | 


2.5 Decision Tree Elicitation From Groups 


255% 1 General. The discussion of Section 2.3 showed that the decision 
making process can be enhanced by using computer techniques of decision tree 
elicitation and aggregation. These techniques include interactive decision 
tree elicitation, sensitivity analysis, utility aggregation, probability 
aggregation, and tree merging. The permit computer-directed "conversations" 
to focus on critical issues relevant to the decision problem at hand. Such 
conversations are guided by information obtained from the individual decision 
maker as he interacts with the elicitation program. The final result is a 
completed decision tree. | 
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| ‘This form of decision aiding can be applied to decision making groups 
in two main ways: 


(1) Complete trees are elicited from all group members; group 
interaction is used to progressively mene and veeo ive the 
individual trees. | 


| (2) A single complete tree is elicited during group interaction; 
_ Immediate individual contributions of all members are merged | 
and resolved at each step of the elicitation process. 


Each approach is discussed separately below. 


2.5.2 Individual. Trees. The first apnroach requires decision. trees from 
each member of the group before discussion begins. Once the collection of 
individual trees has been obtained, a tree merging program begins the 
construction of the group tree by matching the various decision alternatives, | 
event outcomes, utilites, and probabilities. ‘Differences of opinion and oO 


-- conflicts in tree structure detected during the tree merging process are 


resolved in real-time by identifying the elucidating critical issues in the 

group tree and directing group discussion toward these areas. The critical 
issues separate into two major types: structural differences and numerical 

discrepancies, in utilities and probabilities. The resolution of numerical 

discrepancies at a node has been discussed above. Accordingly, we turn our. 
attention to the question of structural differences. 


Structural Differences. A structural difference occurs when one group 
member has alternatives or events in his decision tree that are not present | 
in trees of other members. In these cases, automatic sensitivity analysis 
provides the basis for conflict resolution. The tree merging program has 
the complete individual decision trees of the group members available at all 
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times. It identifies the earliest point (node) at which a structural 
difference occurs, and uses sensitivity analysis to calculate the importance 
of the various decision alternatives at that node with respect to each 
separate tree. Using these importance measures, the computer guides group 
discussion toward selection of a mutually agreeable set of alternatives from 
those occurring on the individual trees. For example, a unique alternative 
found to be relatively unimportant to the associated decision maker js 
suggested for deletion. Conversely, an alternative occurring in most of the 
individual trees, and found to be generally important, is strongly | 
recommended for retention. If two or more group members have widely varying 
opinions on the importance of a specific alternative, the program directs 
the involved individuals to dissect the issue and reach an agreement, as 
described in Section 2.4. The final set of alternatives in the group tree 
will then be a subset of the union of alternatives from the individual 
trees. Other local structural differences, such as disagreements on the 
time-sequencing of events, are addressed in a similar manner. 


2.5.3 Group Trees. The second approach is based on the interactive 
elicitation of a group decision tree from beginning to end during the group 
meeting. As the computer directs the decision making process, the group 
members enter their initial alternatives and numerical value estimates 
individually, and the program supervises conflict resolution at each point | 
in the tree-building process. Sensitivity analysis is used as before to | 
continuously focus on the most critical parts of the tree and to terminate 
discussion when it has been determined that further detail and expansion will 
not significantly affect the overall result. This approach essentially 
treats the entire group as an individual, since only one tree is constructed 
and there is no merging necessary. However, the techniques for conflict 
resolution described above are still required and applicable. 
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2.5.4 Mode Selection. It appears best to select one elicitation mode 

for initial implementation and evaluation. Since they share many required 
software features, and will involve many similar aspects of group decision 
behavior, it is more cost-effective to implement the most practical method 
first, evaluate it extensively, and use the evaluation results for modification 
or progression to the other. Of the two modes, elicitation of a single group 
tree, in which the group is treated as one individual (with occasionally 
divergent opinions), involves the most direct extension of the existing 
tree-elicitation software, the strongest application of sensitivity analysis, 
and the most direct use of multi-attribute utility measurement. Conversely, 
merger of individual trees raises a number of problems, primarily in the 
resolution of extreme structural differences. While solutions to these 
problems appear feasible, more data both about group interaction during aiding, 
and about individual trees, would be helpful. Accordingly, it was decided 

‘to implement the group tree approach during the first year program, but at 

the same time elicit trees from individuals, and examine them for insights 
into optimum merging procedures. — : 


2.6 Benefit Analysis | 


2.6.1 Overview. Evidence from a variety of sources related to group 
decision making shows that significant improvement in the decision process 
can be expected through the use of the proposed aiding technology. This 
improvement can be measured in terms of two major performance factors: 

(1) decision time and (2) decision quality. The next sections provide a 

_ discussion of the benefits that can be expected in each of these areas. 


2.6.2 Reduced Decision Time. Overall reduction in group decision time 
can be expected from three sources: (1) computer guidance; (2) multiattribute 
utility analysis; and (3) sensitivity analysis. 1 | 


Computer Guidance. Computer guidance of group discussions explicitly 


highlights the major areas for consideration. Since the system display 
structures and guides the decision process, the attention of the group 
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members is focused on specific aspects of the decision problem. Thus, there 
is less ambiguity about the topic of conversation and less time 1s wasted 
clearing misunderstandings. 


Multi-Attribute Utility Analysis. Group attention to specific value 
conflicts reduces the noise associated with discussing decision situations. 
This brings the group to a closer agreement about their utility values. 
This fact has been shown by Gardiner & Edwards (1975) in comparing the 
choices of two sets of groups. One set was aware of the attributes of each 
decision choice and the other used strictly holistic judgments. The study 
_ Showed a three-to-one improvement. in the deviation of values for choices 
between sets of groups. In other words, the groups that made choices based 
on multi-attribute utility analysis were three times closer in value agreement 
than the groups that used strictly holistic methods. This improvement 
Clearly indicates that awareness of the attributes of decision alternatives 
‘reduces disagreements dramatically. | 


Sheridan and Sicherman (1977) have recently shown that if a group is 
treated as a single decision maker, and utility measurement is made using 
electronic voting procedures, a complete multi-attribute utility function , 
can be quickly obtained. In accord with our previous discussion of MAUM 
procedures and benefits (Section 2.4) Sheridan and Sicherman state: 


The time requtred to obtain (a utility funetton) from a group 
ts quite small. Depending upon the problem, some minutes of 
discusston are requtred to clartfy the vartables of the sttuattons 
whitch are betng compared. The group does not need to know anythtng 
of the theory. In the two-attrtbute example described only four 
votes were necessary, each of whtch took about two minutes of meeting 
time. The procedure can easily be extended to larger groups or 


sttuattons of 3 or 4 dimenstons. When a group proves to be 
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heterogeneous, so that.calibratton of a utility function ts not 
valid, the process reveals useful bounds for sensitivity analysis 
and specifies the degree to which final dectstons concur with — 


group preference. 


sensitivity Analysis. The use of sensitivity analysis during the 
tree elicitation process saves a considerable amount of time since the © 
computer is continually guiding the group members toward the most critical 
issues. Valuable time is not spent arguing about unimportant topics. 
Although no experimental data is available concerning the exact amount of 
time saving, the sensitivity analysis approach is similar in many respects 
to the alpha-beta pruning technique used to limit search time on game trees. 
It has been shown by Nilsson that using alpha-beta tree pruning allows a | 
tree to be searched twice as deep as would otherwise be peeeiele (Nilsson, 
1971). As Nilsson states: | 


coe. the number of ttp nodes of depth D that would be generated | 
by optimal alpha-beta search ts about the same as the number of 
ttp nodes that would have been generated at depth D/2. without 
alpha-beta. Therefore, for the same storage requirements, the 
alpha-beta procedure, with perfect successor ordering, allows 
search depth to be doubled. - 148) 


Since it has been proven that the use of alpha-beta pruning does not alter | 
the final decision, this storage savings can be alvectly: translated into 
a time savings. | 


For example, consider a decision tree with three branches per node 
that extends four levels deep. The number of tip nodes in such a tree is 
3" = 81. With alpha-beta pruning, the optimal result can be obtained by 
generating only the number of nodes appearing at level 2, that is, 3° = 9. 
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‘Thus, there is a 9-to-1] time savings for this particular tree. With larger 
trees, the savings would be much more. In fact, it increases exponentially. 
This ratio assumes that the complete tree would have been expanded without 
tree pruning. At typical group meetings, the members subjectively prune 
their "tree" during the discussion. However, the issues that are discussed — 
are usually far from the optimal ones. Thus, even if no-time savings is 

- realized by use of sensitivity analysis, the group can be certain that the 
topics of attention are indeed the most crucial ones. 


2.6.3 Decision Quality Improvement. Decision quality is related to the 
rationality of the final decision, in terms of the potential of the selected 
alternatives to provide the highest expected gain. Nickerson and Feehrer 
(1975) have attempted to discriminate between two aspects of quality -- 
effectiveness and soundness: 


Effectiveness ts determined after the facet. The logical soundness 
of a dectston depends on the extent to whitch the dectsion maker's 
chotee of actton ts consistent with the information avatlable to 
him at the ttme the dectston was made, and with the dectston maker's 


oum preferences and goals. (p.170) 


At Perceptronics, a similar differential has been established between 
external system performance measures (effectiveness) and internal decision 
measures (soundness). The current analysis, however, does not warrant so 
fine a distinction, and accordingly both concepts have been used in arriving 
at an overall estimate. Decision quantity improvement is expected to arise 
from two sources: | 


(1) Increased participation of group members. 
(2) Multi-attribute decision modeling. | 


The following is a discussion of each of these aspects. 


Member participation. | Related work in group decision making indicates 
that decision quality can improve by increasing the participation of the - 
group. In a command group which consists of various specialists, optimum 
decisions require all points of view to be considered and all aspects of the 
problem to be evaluated. Thus, if the system can increase the participation 
of the group members it can be expected that decision quality will improve. 


The important effects of direct feedback on individual participation 
and opinion considerations have been shown by Sheridan (1975). Sheridan has 
developed an Electronic Voting and Discussion Technique (EVOT), which 
utilizes an electronic voting scheme and specially designed meeting | 
procedures. Every participatant is able to make an anonymous coded response ~ 
to questions posed by the moderator, and to observe instantaneously a tally | 
of how many members voted in each category. The purpose is to get a rapid 
appraisal of areas of consensus and areas of controversy. This allows 
participants to reveal their ignorance anonomously, to deal with controversial 
questions without intimidation, and generally to make the discussion more | 
responsive to the real interests and needs of the group. The technique has 
been used in configuration with real-time visual feedback amspiey Of a 
purely numerical type. 


For example, in three independent experiments covering race issues 
with groups of 12 to 15 participants, it was found by Pizano (1974) that — 
group participation increased by a factor of 2.5 when the EVOT aid is used. 
For a specific experimental group using EVOT, 90% of the time was used by 
_ Slightly less than 50% of the participants. In contrast, for a control ~ 

group that did not use the feedback, 90% of the time was used by less than 
20% of the participants. (Bearing in mind that in some cases, as much as 
80% of the time was used by one person.) It can be concluded from these — 
experiments that more individuals will contribute to the decision when group 
decision aiding Pec uonegy is PPC ent i 
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Positive effects of group participation on decision quality have also 
been shown in experiments with Delphi techniques by Dalkey & Helmer (1963) 
and by Huber (1974) in studying combined judgment of individuals in decision 
conferences. According to Dalkey, when each group member's view 7s 
genuinely considered ina non-dominating way, the decision quality improves 
with respect to an external criteria (Dalkey & Helmer, 1963). The 
implication from this is, again, that mechanisms which increase the effective 
participating of the group will improve decision quality. Huber (1974) showed 
great sensitivity between (1) the number of individual opinions that are 
actively considered in a group decision and (2) the error associated with 
the decision. His research was concerned with the problem of combining the 
judgment of a number of group members into a single group judgment using 
the Delphi technique. Results have shown that decision error was reduced 
by a factor of 2 when the number of individual participants increased from 
2 to 5 and further decreased by 25% when 5 additional participants were 
included. This indicates greater sensitivity for small groups as, for 
example, may be encountered in command posts. The implication from this 
data is that at small group levels (5 or less) decision quality is expected 
to improve markedly with increased participation. 


Decision Modeling Effect. Multi-attribute decision analysis is 
a major factor which will contribute to decision quality. In particular, 
the use of the multi-attribute model as a criteria to evaluate group utility 
affords a considerable advantage over holistic approaches. Newman (1975) 
analyzed data from different studies and showed that multi-attribute utility 
assessment procedures are more reliable, valid and generalizable than holistic 
assessment procedures. In one study involving allocation of air strike 
missions in a tactical environment (Miller, Kaplan, and Edwards, 1967, 1969), 
a performance improvement ratio of 1.23 was found in favor of the multi- 
attribute approach. In another application concerning group evaluation and 
decision making with regard to land development applications (Gardiner, 1974), 
an improvement ratio of 1.69 was obtained. | 


Significant improvements in decision effectiveness have been observed — 
when interactive computer aiding is used to generate action alternatives. | 
Individual operators in a simulated ASW tracking task (Freedy, Davis, Steeb, 

— Samet and Gardiner, 1976) were found to exhibit 86% greater performance 
effectiveness when aided by a computer based on their observed preferences. 
The improvement was measured by an external criteria which quantitatively 
combined submarine detection score against false alarm rate and cost. — 


The use of the decision model and elicited utilities is another 
major factor that. improves decision quality. The early bootstrapping approach — 
of Dawes (1971) and Goldberg (1970) resulted in up to 20% improvement in 
decision effectiveness, apparently due to a lessening of operator inconsistency 
or "noise" (Fischer, 1972). Similarly, Miller and his associates (1967) 
demonstrated the efficiency of combining human value judgments and machine © 
policy selection in a tactical air command system. The interaction was 
found to result in 90% of the attainable optimum. performance compared to 
roughly 50% of optimum Perrcnmanice by the operators alone, an improvement 


- of 1.8. 


A direct comparison of decision makers with their individual decision 
models has been made possible by research at Perceptronics dealing with the 
: use of adaptive techniques in computer assisted maintenance training (May, 
Crooks, Purcell, Lucaccini, Freedy, Weltman, 1974). The cumulative cost 
of the tests used to diagnose an electronic circuit provided the performance 
measure. For each subject, a comparison is made between his mean cost, 
observed in experimental trials, and the mean cost accumulated by his model, 
tested in computer simulation. The model's costs are on the average 0.70 
of the individual's costs, indicating a performance improvement ratio of | 
1/.70 = 1.43. For the less skilled individuals, the alia offered by 
the model is markedly greater. 
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2.6.4 Effectiveness Measure. A compound measure for the effectiveness of 
group decision aiding can be formulated in terms of the expected improvement 
in the system performance variables. The effectiveness can be established 
by aggregating available research data. The efficiency measure is expressed 


Effectiveness of | : ‘| Decision Decision 


Group Decision = f | Time , Quality 
Aiding Reduction Improvement 


where Decision Time Reduction is in turn a function of: (1) reduction of 
conflict and noise, (2) concentrated discussion in areas where real value 
difference exists, and (3) reduction of tree size and number of argument 
points which are required to reach a conclusion. The improvement in decision 
quality is due to (1) increased participation and (2) decision aiding effect. 
An overall improvement ratio can then be defined as: | 


Decision Quality Decision Timing 
Improvement Ratio ~ Improvement Ratio 


Improvement Ratio # 
Table 3-2 summarizes the improvement ratios estimated by the above sections. 
The data is based on individual experiments where each variable was studied 
independently. This will probably not be the case ina complete aiding 
system. Accordingly, we can reduce the expected improvement somewhat to © 
account for interdependence, and estimate a ratio in the vicinity of 25:1. 


2-3] 


Factor. | 


Decision Time 


Decision Quality 


Improvement Ratio 


~ Sources 


Conflict reduction 


Tree Sensitivity _ 
Analysis. 


Reduction of noise 


Increased 
Participation 


Aiding Effect 


| Improvement Ratios 


(Min-Max) 
3-5 
4-9 

12 - 45 


Table 3-2. EXPECTED DECISION IMPROVEMENT 
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3. TECHNICAL APPROACH 


3.1 Aiding System Description 


3.1.1 Aiding Functions. Figure 3-1 shows in conceptual fashion the decision 
making group in the conference room. Group interaction centers around the 
large screen display (LSD), which provides guidance information and group 
action feedback. The group may be led by a director (D). It is assisted 

by an intermediator (I), whose functions are to coordinate the computer 
interactive process, to control the information flow, and to initiate 
discussion phases, display programs, and data processing via the computer 
access terminal (CAT). Each group participant (P) has a data entry 

terminal (DET) to enter his responses and information requests. Interaction 
around the table is both verbal and display-directed. 


Figure 3-2 shows the basic group decision making process in block- 
diagram form. The process is driven by an existing decision problem that 
requires group interaction to lead to a viable solution. This is shown at 
the top of the figure by the progression from the decision problem through 

- the group decision making process to the problem solution. The interactive. 
process itself is comprised of a cycle of four major aiding steps. These 
basic aiding steps are as follows: 


(1) Node Selection 
(2) Node Expansion 
(3) Value Elicitation 
(4) Tree Analysis 


Each cycle through the four steps corresponds to the complete expansion of 
One node in the group decision tree. At the end of each cycle, the group 
reviews the current progress and determines if more information is necessary 
(i.e. growing the tree by completing more cycles) or if an acceptable 


LSD LARGE SCREEN DISPLAY 
CAT COMPUTER ACCESS TERMINAL 
DET DATA ENTRY TERMINAL 

I INTERMEDIATOR 
DIRECTOR 
PARTICIPANT 


FIGURE 3-1 | 
THE DECISION MAKING GROUP 
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solution exists at the current time. This stopping point can be determined 
by group concensus or can result from sensitivity analysis. For example, 
if it has been found for all tip nodes that minor alterations in value 

will not have an impact on the final decision, the computer can immediately 
recommend that the group terminate its interaction. The aiding steps 
themselves are described separately in the following sections. 


3.1.2 Node Selection. The first step in the group decision making process 
cycle is node selection. During the group interaction, sensitivity analysis 
is used to determine the most critical and important topics for discussion. 
The most critical topic is that which has the most chance, by a change in 
its value, to alter what is currently the most promising solution. It is 
this topic that is brought to the attention of the group so that it may be 
explored in more detail. However, it is always possible for the group to 
choose a mutually agreeable topic for discussion, rejecting the one 
recommended by the system. In this case, the intermediator overrides the 
computer's choice and enters the group choice. The computer will then 
automatically compare the sensitivity value of the chosen topic with its 

own recommendation and comment on the appropriateness of the selection. For 
example, a typical comment resulting from such a criticality comparison 
might be: 


THE VALUE OF THE CHOSEN TOPIC FOR DISCUSSION MUST BE TRIPLED 
BEFORE ANY DIFFERENCE IS MADE IN THE CURRENT TREE. 


Such comparisons will be quite helpful in encouraging fruitful discussions 
and discouraging unproductive ones. The computer output from the Node 
Selection step will generally be a recommendation, in text form, for a 
topic of discussion. The input required is either a confirmation of the 
selection or another desired topic. | 
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3.1.3 Node Expansion. The second step in the aiding process is node 
expansion. During this step, the group is asked to supply a list of decision 
alternatives or event outcomes. This list must contain mutually exclusive 
items, due to the nature of the decision tree formalism. Aid in determining 
satisfaction of the mutually exclusive condition can be provided by the 
intermediator if necessary. 


The interaction between the group and the computer system may require 
a relatively sophisticated response interpretation mechanism, since elicitation 
here is characterised by English text input. The text is composed of keywords 
and phrases that describe the basic decision alternatives or event outcomes. 

It will be necessary to compare labels on branches so that repetitive or 
redundant alternatives can be identified. This process must be oriented toward 
keyword search and matching, since the group decision aiding system is assumed 
_ to be domain-independent. | 


If preliminary individual decision trees are available, they can be 
used as an initial source for decision alternatives or event outcomes 

by subjecting them to a tree merging process. After the location of the 
current topic has been found in each tree, the union of the branches of 

all trees will form a basic set from which to work. If there is a disagreement 
as to node type (i.e. decision or event), this must be resolved by the group 

as a separate issue. 


3.1.4 Value Elicitation. Once the major decision alternatives and event 
outcomes have been established, utilities, probabilities, and costs must 
be determined for each one. Through computer guidance, each alternative 
or outcome is displayed and discussed, one at a time, in order to define 
these values. Utilities and costs are necessary for decision alternatives 
as well as event outcomes; probabilities are necessary only for event 
outcomes. If utilities are given on a relative scale, rather than in 
money, then costs are assumed to be contained within the utility estimate 
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and do not have to be elicited separately. If the group agrees on approx imate 
values immediately, no further processing is necessary. However, the values 

— entered into the individual input terminals may differ greatly. In such | 
cases, a conflict exists and it must be resolved before tree elicitation 

can continue. oe | 


When the computer detects a genuine conflict in utility values, the 
multi-attribute utility model is activated to help resolve the conflict by - 
decomposing the issue into its constituent parts and assigning utility 
values to the parts. Then, an aggregation algorithm determines a final 
value from the decomposed parts. (Section 2.4 describes the procedure for 
recognizing conflict and obtaining multi-attribute utility values from 
groups.) Conflicts in probability assessment and in costs can be handled 

with a similar model. an 


-3.1.5 Tree Analysis. The last major step in the process cycle is a purely 
computational one. With the completion of node expansion, the tree is 
| ready for expected-value rollback analysis to determine the best decision 
with the currently available information. This computation not only allows — 
the group to see the progress of the interaction, it also prepares the © | 
tree for sensitivity analysis in the first step of the next cycle. 


| The rollback is the primary means of determining the best decision 
thus far. The calculation is made starting at the tips of the tree and ~ 
progresses back toward the single root. At each decision node, the maximum 
of the values of the branches is assumed to be the value of the node. This | 
is becasue the decision makers have a choice and would naturally pick the ~~ 
alternative with the highest utility. At each event node, the expected value © 
of the branches is taken as the value of the node (the sum of the products 
of each utility with its corresponding probability). This function is used 
because the decision makers do not have the choice of which event will occur. 


3-6 


The outcome is out of their control. When these calculations reach back 
to the root, the branch with the highest computed utility value is the 
currently best choice to take. 


3.1.6 System Organization. Figure 3-3 shows the planned organization 
of the group decision aiding system. The computer access terminal (CAT) 
of the intermediator and the data entry terminals (DET) of the group 
members are connected to the function control module. This module is the 
system supervisor, which is responsible for controlling the following 
functions: | | 


(1) large screen display output 

(2) terminal input and message routing 
(3) data base access 

(4) group decision making process cycle. 


The large screen display is shown under the control of a display format 
module. This module is necessary since the screen may be divided into more 
‘than one active area. | | | 


The group decision process cycle as described in previous sections 
is shown in Figure 3-3 as a series of separate modules (steps) each with 
a corresponding algorithn. The node selection function uses sensitivity 
analysis. The node expansion function uses tree merging if preliminary 
trees are available from each group member. The value elicitation function 
uses multi-attribute utility analysis if value conflicts arise. Finally, 
- the tree analysis function uses expected-value rollback calculations. Each 
of these functions has access to the data base where the group tree is 
stored. | 


In addition to the four basic process functions, the function control 
module can activate a separate direct access path to the data base. In many 
cases, the group will wish to interrogate the data base to aid in its decision 


| Function | 
Control | 
Display 


@ 
| Format - 
| Control | 


- [intermediator 
{Terminal (CAT) 


Node 
Expansion 
Value 
Elicitation 


Tree 


Node 
Selection 


| Sensitivity 
| | Analysis 


Terminal 1 
(DET) 


Multi- 
Attribute 
Utility 
Analysis 


Expected 
Value 
Rollback | 


- Screen | _ yp nti Pn 
| \ Display | | | Oo a ne _& Bm 


Analysis 


Terminal n 
(DET) | 


FIGURE 3-3 | 
ORGANIZATION OF GROUP DECISION AIDING SYSTEM 


3-8 


task. The intermediator will have a capability to request the display of 
data in varying formats to accommodate the member's requests. 


3.2 System Software 


3.2.1  Qperating System. The following software systems are proposed for 
the implementation of the group decision tree elicitation algorithms: 


(1) UNIX operating system 
(2) LISP programming language 


A version of LISP is available for the UNIX operating system and requires 
a relatively small amount of storage (about 4K). LISP is ideal for 
implementing the group decision aiding system for the following reasons: 
(1) suitability, (2) utilization of existing programs, and (3) transferability. 
LISP is well suited for tree generation and manipulation because of its 
"list-processing" data structure. The existing tree elicitation program, 
written by Dr. Leal, is already written in LISP and may be used almost 
directly. Finally, the LISP system can be transferred along with the 
program source code if necessary. | 


‘Bu2.2 Program Modules. The group decision-aiding system requires program 
modules for each of the four process steps as described in Section 3.1. These 
- modules and their constituent routines include: 


(1) Discussion Control 


(a) A sensitivity analysis module for isolating the most 
critical parts of the tree. | 


(b) A criticality comparison module for determining differences © 
between two suggested topics of discussion. 
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(2) Tree Elicitation 


(a) An elicitation module for directing discussion to the 
| generation of decision alternatives or event outcomes. 


(b) A response interpretation algorithm using keyword search 
- for text comparison. 


(c) A tree manipulation program for maintaining the correct 
structure for the group decision tree. 


(3) Value Elicitation 


(a) A utility scale elicitation package for the definition of — 
individual relative utility scales. (One time use.) 


(b) A probability calibration package for adjusting for 
bias in predicting probabilities. (One time use.) 


(c) A utility and probability elicitation module for determining 
- the required information for proper tree anety eas at ic 


Stage. 


(d) A multi-attribute utility analysis module for conflict 
resolution of estimated values. | | 


(4) Tree Analysis 


(a) An expected value rollback algorithm for determining the 
Se 7st desirable a LECrae eS from the group ree 


(b) A multi-attribute utility analysis module fe conflict — 
~ resolution of estimated values. 


(5) Service Routines - the following service modules are required 
for system use. | 


(a) Display control 
(b) Function control (supervisor) 
(c) Data access routines 


3.0 Systen Hardware | 


3.3.1 Components. Major components of the aiding system hardware are as 
follows: | 


(1) Computer System 
(2) Individual Input/Display Terminals 
(3) Group Feedback Display | 


_ The equipment proposed for each is described separately below. 


3.3.2 Computer System. The group aiding programs will be supported on a 
PDP/11 minicomputer. The computer system, which is currently under order 
by Perceptronics, includes a main frame, disks and controllers, printer, 

DEC graphics terminal, and DEC scopes. The capabilities of this system are 
more than adequate for the proposed application. In addition, the PDP/11 

has become virtually a standard in the ARPA and general military communities, 
thus transferability of the aiding programs will be excellent. 


3.3.3. Individual Terminals. Group member interaction with the aiding 
program will normally be through a DEC Scope VT52 Video Terminal or equivalent. 
The VT52 video display terminal displays 24 lines of upper and lower case 
text at selectable speeds to 960 characters per second, and permits user input 
through a standard keyboard and auxiliary numerical keypad. The terminal, 
_ illustrated in Figure 3-4, is a compact unit reflecting good human factors 
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design, and is well suited to the proposed application. In certain 
experimental situations, as discussed below, the group members’ terminals 

may be replaced by numerical and special-function keypads. It is anticipated, 
however, that the intermediator will always operate from a terminal. 


3.3.4 Group Feedback Display. It is planned to provide a centralized 
large-screen color display for feedback of information needed by the group 
as a whole. The display system, shown schematically in Figure 3-5, is 
composed of three components: 


(1) DEC PDP/11 Minicomputer 
(2) DEC VT30 Display Controller 
(3) Advent 750 Video Projector LSD 


The PDP/11 Minicomputer has already been described. The other two units 
are described separately below. Use of color in the display system adds 
little to the cost, since standard 525-line video is employed throughout, 
but it adds considerably to the graphic impact and flexibility of the 
feedback displays, allowing color-coding of critical display contents, and 
separation of individual responses. | | 


Large Screen Display. Presentation of the feedback display will be 
done through the Advent Model 750 Videobeam Color Television Projector. 
Figure 3-6 presents the major components and specifications of the Advent 
Model 1000A professional unit. The Model 750, just out on the market, is 
a consumer version of this unit, with a smaller projector case, slightly 
smaller screen (60" x 45"), and essentially the same TV specifications. It 
is designed to sell for $2500 rather than for $4000, the cost of the Model | 
1000A. Both units offer high quality, full-color video, projected ona 
bright screen intended specifically for group viewing. Both units can be 
driven from conventional color video signal sources. | 


DEC PDP 11 
MINICOMPUTER 


DEC VT30 
COLOR VIDEO 
DISPLAY CONTROLLER 


ADVENT 750 
COLOR VIDEO 
PROJECTOR 


LARGE SCREEN DISPLAY 


SPECIFICATIONS 


Picture Size | 


Al, ft. h. x 54/5 ft. w. (132 cm h. x 178 cm w.); 7-foot diag- 
onal (2.2 m); 24.5 sq. ft. (2.3 sq. m). Not variable. 


Brightness 


More than 20 foot-lamberts (above the brightness range 
recommended for motion picture theaters by the Society 
of Motion Picture and Television Engineers). 
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VIDEOBEAM RECEIVER-PROJECTOR. 


. Audience Size 


With screen and projector sitting on the floor as illus- 
trated, approximately 50 viewers can be accommodated, 
depending on room size and configuration. By raising 
both screen and projector according to instructions 
supplied on request, coverage can be increased to 200 
or more viewers. 7 


Screen to Projector Lens Distance 
100” = 1” (254 cm + 2.5 cm). Not variable. 


Projection Tubes 


Each of the three Advent-developed and manufactured 
LightGuide projection tubes* incorporates a wide-angle 
Schmidt projection system. A spherical mirror sealed 
inside the tube collects the light emitted from a phosphor- 
coated aluminum target and projects it to the screen 
through a molded acrylic aspheric lens bonded to the 
tube. 


Opticalspeed . . . . f.7 
Mirrorsize . . . . . 6" (15.2cm) 
Target size . . . . . 3.3" diagonal (8.4 cm) 
aE VIR EOSERM SCAEOS: Anode voltage. . . . 30 KV 
a Beam current . . . . 200 microamperes 
maximum per tube 
Deflection . . . . . @8degrees per tube 
Focus . . . . . . - Primary permanent 
magnet with secondary 


electromagnet for 

incremental focus 

adjustment by user 
Resolution | 


Determined by NTSC video bandwidth; not limited by 
'- electron optics, projection lens system, or segmentation 
of raster into color dots or stripes. 


Signal Standard 


VideoBeam television operates from regular American 
NTSC broadcasts and external video sources; 525 scan- 
ning lines interlaced 2:1 with 60 Hz field rate and 30 Hz 
frame rate; picture aspect ratio 4:3. (Sets for use with 
other signal standards are available outside the U.S; 
contact Advent for further information.) 
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Display Controller. The proposed video signal source is the DEC 
Model VT30 display controller. This unit was recently developed by DEC's 

| Computer Special Systems (CSS) group, working from an earlier version 
available only in Europe. It attaches directly to the PDP/11 bus, and 

| generates the video signals necessary to proces Baconer alphanumerics 

and graphics. | | 


Figure 3-7 is a block diagram of the controller unit. For graphics | 
_ purposes, the Monitor screen is divided into 2304 squares arranged in 36 
rows of 64 characters each. Each square consists of an 8 x 8 dot matrix. 
In order to display graphic information, the squares are filled with the | 
required characters. The Display System allows the user to define its | 
own set of special characters. Up to 64 characters, specified as particular 
bit patterns within an 8 x 8 matrix frame, can be defined dynamically, 
by loading a Read/Write Memory within the Controller. Dynamically means 
that any character can be changed at any time under program control. In 
addition to the 64 user- -defined characters, the 64 upper case ASCII characters © 
are also available as standard 5 x 7 matrix from a ROM Character Generator. 


Pigune 3-8 illustrates the Controller's graphics anneal. As 
seen, this unit is excellent for the display of decision trees, graphics, 
- bar charts, situation maps, and related alphanumeric information. The 
controller is priced at $7500. This is low compared to the cost of 
equivalent color graphics controllers available from such firms as Video| 
Disk, Ramtek, etc. According to the CSS group, relevant graphics software 
is available in the DEC library, and they themselves will assist early 


users to develop special graphics routines. 


3.3.5 Group Aiding Facility. Figure 3-9 shows the planned configuration 
of the large-screen group display and individual interactive terminals in 
an existing room of Perceptronics' Woodland Hills facility. The Advent © 

~ Model 750 projector, which is only 26" high, will fit under a standard 
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conference table. Group members, including the intermediator, will be 

seated around the table in front of their terminals, and will be located 

in the display's prime viewing area. Sufficient space is still available 

for experimental observers. Alternatively, if it appears desirable to 

remove observers from the immediate discussion area, a view-window can. 

be cut in the back wall, and the observers located (behind glass or one-way 
mirrors) in the adjacent room. | 2 | 


3.4 System Evaluation 


3.4.1 Major Aiding Factors. A number of aiding factors can be expected 
to impact the type of interaction that the group has with the aiding system, 
and its decision making performance. These factors include: — 


(1) Tree Elicitation Mode. Two major elicitation modes have been 
identified. In one, the tree is elicited from the group as a 
whole. In the other, complete individual trees are merged by 
group action. Elicitation from groups could: (1) minimize 
points of conflict, (2) use sensitivity analysis to explore 
only branches important to the group as a whole; (3) simplify 
system implementation, including programming and presentation. 
On the other hand, elicitation from individuals might (1) 
produce more unique decision alternatives, (2) allow more 
flexibility in personnel scheduling, and (3) allow the group | 
to identify general areas of agreement beforehand and therefore 
focus on only major problem areas during interaction. Preliminary 
analysis suggests that group elicitation is most advantageous — 
but because of the potential benefits, a more detailed 
comparison with individual elicitation seems warranted. 


(2) Intermediator's Role. The notion of a so-called "intermediator" 
to intercede between the members of the decision making group 
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and the decision aiding program has been very well received 
when presented to command level personnel. An attractive 
feature is that it could relieve the actual decision makers of 
almost all responsibility for learning "hands-on" operation | 
of the aiding system. At the same time, it places a relatively 
heavy load on the intermediator himself. Thus, one might 

ask whether he is in fact essential; what is the tradeoff 
between his functions and those of the group members; and 
correspondingly, what should be the format of the interactive 
programs with which each 1S in contact. 


(3) Interactive Configuration. As an adjunct to the above, one 
can conceive of a broad spectrum of input/display mechanisms 
for the intermediator and the decision makers. At one extreme, 
the intermediator would have full interactive capabilities 
with the program, while the decision makers would have only 
numerical entry pads, or even simpler switches or potentiometers 
for entering utility and probability values. At the other 
extreme, everyone would have computer terminals able to 
interrogate the program independently, and could follow up 
points of personal interest even during the course of group 
interaction. Clearly, it would be impossible to implement all 
variations in this area, but some attempt to match requirements 
to the characteristics of actual decision making groups, at 
several command levels, appears worthwhile. 


(4) Presentation Formats. Careful human factors analysis and design 
will be necessary to optimize the presentation of the key feedback 
factors to the group. In particular, the intragroup conflicts 

arising during tree elicitation must be clearly represented. 
Moreover, representation of conflict should be done in a manner 
which points toward its resolution. It is anticipated that 


development of display formats will proceed in an evolutionary 
manner, with initial choices modified following actual group 
experience. Fortunately, the display of program data, while 
highly critical, is usually more easily modified than are 
the more basic program functions. - | 


(5) Decision Problem Area. The type of decision problem addressed 
by the group can be expected to affect its interaction with 
the aiding system. Since the system is directed toward 
elicitation of trees and resolution of tree-related conflict, 


one would anticipate its greatest benefits in aiding decisions 


for which there are a good number of options, the scenarios 
are relatively complex, and the outcome values are largely 
judgmental. To test this hypothesis, it would be of interest 
to examine computerized group aiding for representative 
command-level decisions which differ on one or more of these 
dimensions. For example, a meaningful comparison could be 
made between two decision types of equal complexity, one of 
which has outcomes whose monetary values are directly known, 
the other of which has outcomes measurable only in relative | 
utility terms. a | _ 


(6) Group Composition. The make-up of the decision making group : 
itself should also have a strong influence on aiding system 7 
performance. A major factor may be previous familiarity 
with computer aiding per se. Perceptronics has found in 
other studies that careful indoctrination into the goals 
and function of a decision aiding system significantly 
improves system performance. Whether a group is like-thinking — 
or highly divided regarding the general approach to a decision 
problem would also affect how it uses the computer-aiding 
capabilities. Level of command , and the associated decision 


styles involved, will further color the group-computer dialogue. 
Group size should not be as significant a factor under aiding 
conditions as it is normally, because the computer essentially 
handles all individual inputs simultaneously, and larger 

groups may actually come more rapidly to a consensus. Exploration 
of these factors could be done economically by allowing varied 
groups, of reasonably small size, to encounter the same types. 

of problems under controlled conditions of system indoctrination. 


3.4.2 Methodology. Perceptronics plans to conduct empirical tests of the 
group aiding system in accord with the following methodology. 


Facility. One room of Perceptronics' new office area will be 
designated the group decision facility, and will be configured as illustrated 
in Figure 3-9. This facility will be used for subject briefings, for system 
explanation, and for actual test sessions. 


Subjects. It is planned to work initially with decision-making 
groups of three to five members. Group subjects will be selected to be as 
representative as possible of personnel who would interact with such 
systems in a military setting. Although it will be difficult at this program 
stage to obtain active-duty, senior-level military personnel, Perceptronics 
has had good success in the past using reserve officers from local Naval 
units. It is also planned to use university graduate students for 
comparison purposes. But initially, the test groups will be kept as 
homogeneous as possible. 3 


Procedure. Section 1.2.3 has outlined the reasons for our choice 
of counter-terrorist activities as the initial problem area for group 
decision making. Several suitable problem scenarios in this area will be 
developed early in the first-year program, and supporting documentation 
will be produced. Subject decision making groups will first be familiarized 


with the general area of decision trees and decision analysis. They will 
then receive a demonstration of the aiding system, given by an intermediator, 
and will be led through one or more brief sample problems. This will occupy | 
one or two meetings. During the actual test session, they will be introduced 
to the problem scenario, given the supporting documentation, allowed to 
study it briefly as individuals, and then convened as a group for a : 
problem-solving meeting scheduled for about two hours. 


Performance comparison between aided and unaided decision making 
groups raises both theoretical and practical problems. Foremost is the 
question of how the aided group would approach the decision problem. If | 
the group is allowed to proceed in a completely unstructured manner, problem 
formulation and group dynamics would differ completely from the aided case. 

_ For example, emergence of strong group leader, who dominates discussion, 

might radically shorten solution time at the expense of analytical breadth 

and depth. On the other hand, if the unaided group is asked to attempt | 

a "manual" decision analysis, then a trained analyst would have to be | 
assigned to it, and the lengthy procedures of interview, utility measurement, 
etc. would have to be carried out for each control case. While this holds 
some interest as a means of establishing a state-of-the-art performance 
baseline, it would seem a misuse of program resources to attempt enough | 
manual analyses to yield a statistically significant comparison. Accordingly, 
we plan at present not to conduct a formal test of cases; but if it proves | . 
feasible, we will conduct one or more manual analyses to obtain a aren? 
comparison standard. | 


Measures. Experimental data will be gathered during the test sessions 
for the purpose of analyzing the group interactions, characterizing the | 
group's decision performance, and assessing the acceptance of the various 
aiding functions. Selection and formal definition of measures will be 
done in an early program stage. Those under consideration include: 
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(1) Problem Solving Protocols. Closed-circuit TV recording can 
be used to provide a detailed audio-visual record of interactive 
processors during problem solution. Since the group display 
will be video-based, an effects generator could be used to 
split the videotape between the current display and the 
groups response to it. Perceptronics presently has a TV 
camera/recorder/monitor system suitable for this purpose. 


(2) Decision Time. Overall and elemental decision times will be 
directly available from the computer program, and can be 
printed out as a summary report following the test session. 


(3) Decision Performance. Since the groups’ decision will not 
actually be implemented, soundness of choice will be the 
main performance criterion. Number of alternatives raised 
and considered will be available from computer and video 
records. Data on the extent to which the group acts in accord 
with its own values (achieves maximum expected utility) will 
be available from the sensitivity analysis portions of the 
program. In addition, comparisons can be made between 
utilities and decisions of the test groups and those of 
recognized experts in the problem area. 


(4) Participation. This factor has been identified as important 

in the preliminary effectiveness analysis. Computer-directed 
participation of each member will be recorded by the interactive 
program. Associated verbal participation will be available 

from the video recording. 


(5) Subjective Response. Group debriefing, selected individual 
interviews, and individual questionnaires will be used to 
determine the attitude of the test subjects toward the 


computerized aiding system. Of particular importance will 
be questions regarding: problem clarification, individual 
inclusion, depth of analysis, role of intermediator, types 
of interaction, methods on conflicts resolution, and 
confidence in eventual solution. 7 | 


4. PROGRAM SCHEDULES 
4.1 Overview 


The general objectives of the proposed R&D program were stated in 
Chapter 1. Subsequent chapters outlined research accomp] ished to date 
and discussed in some detail the planned technical approach. This chapter 
deals with the means by which the proposed program will be carried out. 


4.2 Two-Year Plan 


A two-year R&D schedule is proposed with the following division of 
major program goals: 


(1) First Year -- Model development, prototype system implementation, 

| system concept demonstration, pilot experimental evaluation 
performance measure analysis, system description and evaluation | 
report. | | 


(2) Second Year -- System expansion and modification as. required. 

| Full scale experimental evaluation, using additional decision 
problem areas. Preparation of software in transportable 
form. Development of guidelines for future military application. 
Final report. 


ALS Proposed First-Year Program 


A four-phase effort is proposed for the twelve month period. The 
phases are: | 
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(1) 
(2) 
(3) 
(4) 


Planning and Design Specification 
System Development and Integration 
Experimental Evaluation Studies 
Analysis and Reports | 


The following presents the main features of each program phase. 


4.3.1 


will:. 


~—6©4.3.2 
will: 


(1) 


(2) 


(3) 


(4) 


Planning and Design Specification. During this phase Perceptronics 


Specify and develop problem scenarios for group decision making. 


Develop algorithms for value conflict identification and | 
resolution using multi-attribute utility analysis, establish 
procedures for tree merging and formation of group decision 


trees, and adapt the existing tree elicitation and sensitivity © 
analysis programs to the context of group decision making. 


Develop and specify the required display programs and formats 
for presenting information to the group and establish a 
methodology for representing the areas of conflict. Define 
procedures for group computer interactions. 


Design system software, specify all required programs and | 
subprograms, data files, and interactive functions. 


System Development and Integration. Dur ing this phase, Perceptronics | 


(1) 


Design the computer subprograms for: (a) tree elicitation, 
(b) utility and probability elicitation, (c) tree sensitivity 
analysis, group tree formation, and expected value rollback, — 
(d) group/individual interactive protocols. and (e) display 
control functions and data access. | 
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4.3.3 
will: 


(2) Code all software, integrate complete system on the PDP-11 
hardware. 


Experimental Evaluation Studies. During this phase Perceptronics 


(1) Perform evaluation test to verify the various operational 
features of the system and modify as required. 


(2) Design and conduct a set of evaluation experiments using 
representative decision making groups to examine the effect. 
of group aiding on group behavior and decision performance — 
in the area of counter-terrorist activities. 


(3) Use the decision aiding system to elicit decision trees from | 
representative individual subjects for comparison with the 
group product. 


(4) Perform analysis of the problems involved in merging complete 


individual trees, incorporating both theoretical considerations 
and the results of the group individual comparisons. 


4.3.4 Analysis and Report. During this phase Perceptronics will: 


(1) Analyze the group and individual experimental data and document 
- the experimental results. 


(2) Prepare a year-end technical report including a description 
of the system concept, specification of system programs, 
experimental method and results, preliminary conclusions, 
planned future research, and recommendation regarding the 
application of the group decision aiding to military operations. 
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4.4 Milestone Chart 


Table 4-1 is a Milestone Chart showing the ‘planned ge schedule 
for the initial twelve vent period. 


4,5 Personnel Schedule 


| | Table 4-2 is a personnel schedule in which the hours budgeted 
for the project staff is distributed over the main program phases. The 
proposed personnel level is approximately 2.5 people- ae nouau ene over 
the 12-month period. | 
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. System Development 


2.2 Coding & Integr. 


. Evaluation Studies — 


; Report. 


PROGRAM TASK 


. Planning and Design 


1.1 Problem Scenarios 
1.2 Tree Algorithms 

1.3 Display Algorithms 
1.4 Software Design 


2.1 Subprogram Dev. 


3.1 Oper. Test 
3.2 Group Tests 
3.3 Ind. Tests 
3.4 Tree Merge eet 


4.1 Analyze Data 
4.2 Prepare & Produce 


0 1. 


Months ARC | 


/. TABLE 4-1. 
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PROGRAM MILESTONES 


TABLE 4-2. PERSONNEL SCHEDULE 


System Prototype — Eval. Analysis Total © 


Design Development Studies & Report 
Senior Scientist 250 200 200 | 150 ~ g00 | 
Engineering Psychologist 150 150 250 200. 750 
Senior Engineering Analyst 250 | 150 150 — 90 —600 
Programmer | , 250 — 1000 2000 | 1,500 | 
Research Assistant | | | | | 400 | — 100 = 500 
Consultants 150 50 50 300 


4,450 
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5. MANAGEMENT AND QUALIFICATIONS — 


5.1 Corporate Background - 


Perceptronics is a California corporation, incorporated in Los 
Angeles on April 14, 1969. The company is directed by applied scientists: 
Dr. Gershon Weltman, President, and Dr. Amos Freedy, Vice President. 
Their goal is to apply advanced concepts in informational, behavioral, 
and computer sciences to the solution of real-world problems. The company 
supplies R&D studies and specialized hardware/ software systems to customers 
in industrial, defense, and space applications. 


Perceptronics specializes in three related technical areas. One 
is decision making, which includes the use of adaptive and artificial 
intelligence computer techniques for automation, aiding, and training of 
complex decision processes. The second is advanced automation, or | 
-. robotics, in which computer technology is used to provide autonomous machine ~ 
operation as well as new display/control techniques. ‘The third is human 
factors engineering, which includes human factors analysis and design, 
training, organizational effectiveness, and biomedical systems. 


In addition to its main facility in Woodland Hills, California, 

_ Perceptronics maintains branches in two other locations. The Decision 
Research Branch, Eugene, Oregon, focuses on basic and applied problems in 
the psychology of judgment and decision making. Perceptronics Israel 

Ltd., Tel Aviv, Israel, provides human factors and software services to the 
IDF and other local agencies. 


Perceptronics' staff includes senior personnel in the areas of 
engineering, applied science, and psychology. They, and the other talented 
project people, have had broad experience in both laboratory research and 
in the design and installation of working systems, as well as in the 


presentation of results to scientific and technical audiences through 
journals, reports, and professional meetings. Perceptronics staff members 
have together contributed several hundred technical and scientific 

- publications in their special areas er work. 


5.2 Project Organization and Personnel 


Perceptronics project management is designed to perform three 
main functions: (1) insure top-level scientific and technical direction, 7 
(2) provide day-to-day activity review, (3) take full advantage of skilled 
company personnel. Table 5-1] is an organization chart showing the corporate 
division by major technical group. Each project is assigned to a technical 
group. The main elements of project organization is as follows: | 


Program Director. The Program Director is responsible for overall 
administrative control and direction. He supervises the proposal effort 
and the budget planning and insures compliance with major contractual 
| requirements. Dr. Freedy or Dr. Weltman generally performs this function. 


Group Director. The Group Director coordinates projects within a 
given technical area, coordinating personnel assignments, scheduling of 
facilities, etc., and helping to maintain consistent scientific and 
technical — 


| - principal Investigator or Program Manager. The Principal _ 

Investigator (Program Manager) is responsible for the scientific (technical) 
progress of the project and for executing the details of the research and/or. 
development plan. He supervises directly the project team and support | 
personnel. | | > 
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Project Teams. Perceptronics project teams are made up from the 
research and technical staff on the basis of optimum contribution to the 
project goals. Teams generally contain three to five members. These 
usually include a technologist, engineering psychologist, computer analyst, 
One or more programmers, a research aide, and technical support. | 


Contracts and Accounting. Contractual details are handled by the 
Director of Administrative Services. Billing and cost feedback are 
provided by the Business Manager using a full-time bookkeeper and the 
corporate accountant. The objective is to maintain tight control! of contract 
budgets and obligations without overly burdening the project's scientific 
and technical people. | | 


Resumes. Resumes for key project personnel are provided in 
Appendix A. | 


5.3 Facilities 


Plant. Perceptronics occupies about six thousand square feet in 
a modern air-conditioned building in the Warner Ranch Industrial Park, | 
Woodland Hills (the San Fernando Valley portion of Los Angeles). The 
well-lit office area provides comfortable desk space for the administrative, 
scientific, and engineering personnel, along with conference and library 
facilities. There are separate enclosed areas for the computer facility 
and interactive display systems, and for electronic test and assembly. 
These areas have sufficient room for operation of robotic and manipulator 
devices. | 


The Perceptronics’ facility in Eugene, Oregon, occupies approximately 
thirty-five hundred square feet of office space in a modern building situated 
close to the University of Oregon, whose library, computing, and other facilities 
it regularly draws upon. The facility has space for subject experimentation | 
and for operation of a small computer. | | | 
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Computer Facility. Perceptronics maintains an inhouse computing 7 
facility built around two Interdata Model 70 minicomputers with 24K of 16-bit — 
memory capacity. The third-generation, high-speed {7 usec) machines © 
combine substantial computing power with small size and low operating costs. 

The Interdata Model 70 utilizes Assembler and Fortran IV programming languages: 
Computer system accessories include the following: | | - 


IDIgraf Graphics Display Terminal 
ADDS Consul 580 CRT Terminal _ 
Interactive 3-D Display Terminal | 
Centronics 306 Printer — 

Pertec Dual Disk Drive 

+I/0 Modular Interface 


_ The Perceptronics’ facility is ideal for experimentation with | 
‘interactive man-machine systems and for control of remote manipulators and 
other devices. For large-scale off-line data processing, Perceptronics a 
maintains an active account with the PRC computer center, a short distance 

from Perceptronics’ plant. 


~—6L4 Previous Contract Experience 


A number of previous and ongoing ne srejects are summarized 
on the Following pages to demonstrate Perceptronics’ breadth of contract . 
experience. Perceptronics has shown its ability to adhere to firm schedules: | 
of achievement and cost on all of its previous programs. | 
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1. ADAPTIVE DECISION MODELING 


1-01 Man/Machine Interactions in Computer-Aided Decision Making and Control 


| This study program was undertaken for the Office of Naval Research to 

establish human factors criteria for computer aided decision making and 
control. The program focuses on optimization of the man-machine interface 
-- what information to give the human operator, and how to best allocate 
system functions between man and intelligent machine. 


Contract: N00014-72-C-0093 | 
Price: $210,037 Type: Cost plus Fixed Fee 
Period of Performance: 15 November 1971 -- 15 November 1975 


1-02 Adaptive Manipulation Program | 


Working with Jet Propulsion eaberatory for NASA, Perceptronics | 
developed and installed an adaptive computer program to guide a remote 
manipulator in sample collection on a lunar or planetary surface. The 
program learned sample-col lection Strategies from a human operator, and 
enabled the remote manipulator to improve —— time under contro] 
time delays. | 


Contract No. NAS-7-100 | | 
Price: $16,000 Type: Fixed Price Subcontract 
Period of Performance: 1 March 1972 -- 1 July 1972 


aan 


1-03 Adaptive Computer Aiding in Dynamic Decision Processes 


Perceptronics was awarded an ARPA funded and ONR managed contract to 
Study adaptive computer aiding systems for human operators in dynamic 
decision processes (i.e., intelligence data gathering tasks). The work 
involves the development and evaluation of adaptive methods to facilitate 
and improve human decision making, as well as unique display formats. 


Contract No. N00014-73-C-0286 | 
Price: $378,600 Type: Cost plus Fixed Fee — 
Period of Performance: 1 March 1973 -- 28 February 1976 — 


1-04 Adaptive Decision Aiding in Computer Assisted Instruction 


This is a research contract with the Army Institute of Behavior and 
Social Science to investigate the application of adaptive decision aiding | 


systems to computer assisted instruction. The Perceptronics approach 


Involves the on-line modeling of a student’s value structure in order to permit | 
instruction in higher cognitive and judgmental tasks. | 


“Contract No. DAH C19-75-C~0013 _ 
Price: $156,397 _ Type: Cost sis Fixed Fee 
Period of Performance: | 2 January eee -~- 31 December 1975. 


1-05 Adaptive Engagement Logic and Control Study 7 


- Perceptronics under subcontract to TRW Systems Group was tasked to © 
select techniques and design algorithms for the application of adaptive 
decision making and artificial intelligence to on-line ballistic missile © 
defense. Initial application of problem solving programs proved highly 
successful, in simulation tests. The study was funded by the U.S. Army 
Ballistic Missile Defense Advanced Technology Center, and is classified 
SECRET. oe | 


Contract No. DASG60-76-C-0037. a | oe 
Price: 93,224 , | Type: Fixed Price Subcontract 
Period of Performance: 1 April 1975 -- 30 May 1977 | 


1-06 Ultrasonic Signa] Processing 


NASA's George C. Marshall Space Flight Center contracted with Perceptronics 
for the development of signal processing algorithms for ultrasonic detection of 
coal seam interfaces. Pattern recognition and other computer techniques are 
used ale increase the amount of information available from retuunang ultrasonic 
signals | 


Contract No: wane | | | 
Price: $28,960 Type: Firm Fixed Price | 
Period of Performance: == ~—~—s*Xi18«~December ‘1975 aoe — 1976 | 


1-07 Adaptive Distribution of C3 Information 


| This program 1s sponsored by the Advanced Research Projects Agency 
(ARPA). Its purpose is to utilize the type of on-line adaptive models 

developed by Perceptronics for the effective and timely distribution of 

information in systems for command, control and communication (C3 

Because information overload is a critical factor in modern computer-based 

C3 systems, adaptive distribution should have a major effect on iis yovel 

effectiveness. 


Contract No. MDA903-76-C-0241 | : | 
Price: $56,516 Type: Cost plus Fixed Fee 
Period of Performance: | 1 March - 30 September 1976 


1-08 Decision Aiding in Anti-Submarine Warfare 


This program represents the initial application to a real-world . 
decision making situation of sg, decision aiding techniques previously 
developed by Perceptronics (see 1-03). Sponsored by the Office of Naval 

Research, the program focuses on the decision environment of the Navy's 
P-3 ASW aircraft. Its goal is to analyze the decision tasks involved, 
develop and evaluate specific aiding techniques, in simulated conditions, 
and provide computer software which can be tested in the aircarft itself. 


Contract No. (In process) EY ee 
Price: $65,820 | Type: Cost plus Fixed Fee 
Period of Performance: 15 April - 15 December 1976 


2. HUMAN FACTORS AND BIOMEDICAL — 


2-01 Patient Monitoring Equipment Manual System 


Perceptronics performed a Human Factors analysis of instructional | 
needs for operating patient monitoring equipment. The work was performed | 
under a service contract to Spacelabs, Inc., Chatsworth, Ca. Specifica- 
tions for operating and troubleshooting manuals were made. These were - 
characterized by modular organization of the manual text and by compatible — 
nomenclature and graphics. A complete set of manuals was eevercree ane 
produced. They are currently in use. 


Contract No. Spacelabs, Inc. P.0. 20800 : 
Price: $16,000 | Type: Fixed Price _ 
Period of Performance: 1 July 1970 -- 31 October 1970 


2-02 ECG Analysis Program 


Under subcontract to Computer Applications, Inc., Perceptronics 
designed a computer program which analyzed ECG signals telemetered from 
subjects during air rescue operations. The program was installed for NEL 


at El Centro Navy Base as part of the Biomedical Data il ia and Reduction 
System. 


Contract No. NOO123-70-C-0145 ee hE 
Price: $9,750 Type: Fixed Price Subcontract | 
Period of Performance — (27 duly 1970 -- 1 January 1971 


2-03 Physiological Data Analysis Program 


| Perceptronics developed and installed a biomedical program for the 
on-line acquisition and analysis of physiological data from an exercising 
infantryman for the U.S. Army Combat Effectiveness Test Facility, Camp : 
Pickett, Virginia. The biomedical program performs safety alarm generation 
and data Storage, and generates a post-run summary report. Under an 
extension to this contract Perceptronics provided human factors and bio- 
engineering support to Spacelabs, Inc. in the development of a radiosonde 
method for measuring temperature in an exercising infantryman. The "radio- 
pill" was successfully tested in an alternate deep body Mempenecine/ sensing 
technique. | | 


Contract No. DAA605-70- C-0884 _ & 3 a a 
Price: $20,538 3 Type: Fixed Price Subcontract 
Period of Performance: _ 22 July 1970 -- 14 October 197] 


2-04 Physiological Telemetry System 


Perceptronics developed and successfully installed an on-line computer 
system for physiological data acquisition from an exercising infantryman. 
The system is used by the U.S. Army Human Engineering Laboratory at Aberdeen 
Proving Grounds. The system monitors three vital physiological indices: 
heart rate, body temperature and skin temperature. Monitoring is by means 
of radio telemetry. The computer generates on-line safety alarm indication 
as well as test data summary reports, combining physiological and timing 
measures. 


Contract No. DAAD05-72-C-0238 | a 
Price: $33,320 Type: Fixed Price Subcontract 
Period of Performance: 1 May 1972 -- 15 October 1972 


2-05 Automated Graphics Testing 


Perceptronics developed a procedure which utilizes human factors 
principles for testing commercial graphics design. Part of this work 
involved the development of SAGE, a computerized system for Automated 
Graphics Evaluation. A-minicomputer operates in a real time mode, con- 
trolling and evaluating a visual test which gives the graphics designer 
timely data on the relative discriminability of complex symbols. An Inter- 
data Model/70 minicomputer was used for graphics display and data acquisi- 
tion. The work was performed under subcontract for the sa 
Corporation of Hawaii. | 


Contract No. Helgesson Design P.O. | | hon 
Price: $12,000 Type: Fixed Price Subcontract 
Period of Performance: 1 December 1971 ~- 30 March 1972 | 


2-06 Human Factors Field Instrumentation Package Development | 


Perceptronics, under contract to the U.S. Army TECOM HQ, Aberdeen | 
Proving Ground, developed a Human Factors Instrumentation Package for use 
in equipment test and evaluation at the various TECOM field activities. 
Neatly packed into two compact field cases are a complete set of ‘instruments 
for environmental and performance measures. In addition to the selected 
measurement equipment, the package includes specialized operating and 
maintenance manuals, and a complete training course. . 


Contract No. DAAD05-73-C-0793 
Price: $84,813 | Type: Cost plus Fixed Fee 
Period of Performance: — 20 June 1973 -- 1 April 1974 


2-07 Human Factors Field Instrumentation Package Delivery © 


As a follow-on to the above R&D contract, Perceptronics was chosen to 
supply four (4) full Instrumentation Packages to selected TECOM Test 
activities. The packages include several equipment items manufactured by | 
Perceptronics, and continues to be ordered by Government agencies. 


Contract No. DAADO5-74-C-0367 | — os 
Price: $99,975 = Type: Fixed Price Procurement 
Period of Performance: 21 June 1974 -- 28 August 1974 


2-08 Nurses Information station © 


Perceptronics provided the design specifications fora <eomouter 
oriented Nurse Information System under a service contract to Spacelabs, 
Inc., Chatsworth, Calif. The system uses advanced minicomputer technology 
to handle record keeping and patient data acquisition at a hospital nursing 
Station. Individual stations can be linked to form a avconpbete hospital 
information system. | 


Contract No. Spacelabs P.0. 31755 
Price: $8,000 —-—s- Type: =Fixed Price > 
Period of Performance: | 1 May 1973 -- 1 aenins 1973 


2-09 TV Requirements for Space Shuttle Manipulator — 


Perceptronics under subcontract to RCA Astro-Electronics Division was _ 
tasked to perform human factors analyses and establish simulation requirements. 
in a Hee abone ches study of W requirements for space Shuttle manipulator 6 
control. | ) 


Contract No. NAS 9-14266 _ : eee 
Price: $32,500 os Type: Fixed Price Subcontract 
Period of Performance: = = 1 duly 1974 -- 2 July 1975 


2-10 Guide to Human Factors Testing 


The Army's Human Engineering Laboratory (HEL), Aberdeen Proving Ground, 
Md., contracted with Perceptronics to produce a guide for obtaining and analyzing 
human performance data in a material development project. The purpose of the 
program is to explain and demonstrate the use of Data Item 1334 to future 
contractors and project managers. It involves the performance of two carefully — 
conducted human factors tests, one on a system in its early eevelpaent phase, 
the other on two systems in a post- 7 phase. 


Contract No. DAADO5-76-C-0734 | | 
Price: $50,000 , ‘Type: Cost Plus Fixed Fee | 
Period of Performance: | 17 November 1975 -- 31 July 1976 — 


3. ADVANCED AUTOMATION 


3-01 Space Station and Shuttle Manipulator Design 


Perceptronics, working with MBAssociates for NASA, was responsible 
for the computer systems design of the space station and space shuttle 
manipulator. On the same project the company developed and installed on 
an Interdata Model 4 minicomputer a unique Trajectory Following Control 
Program (TFC) for generating slow and smooth trajectories with large space 
manipulators. The system was demonstrated with the MBA-NAT manipulator 
arm. | 


3-02 Advanced Underwater Manipulation ; 


This is a research and development contract to design, fabricate, and 
evaluate a computer system for improved control and display in underwater 
manipulation. The work is based on a proprietary computer imaging technique. 
Its objective is to examine new solutions to the problem of controlling 
manipulators under limited viewing, including zero visibility. 


Contract No. Cannot be disclosed 7 3 
Price: $396,418 - Type: Cost plus Fixed Fee 
Period of Performance: 1 February 1973 -- 31 June 1975 


3-03 Manipulator Control System. 


Perceptronics designed, fabricated and installed a computer-oriented 
manipulator control system for the Guidance Technology Group of the Jet | 
Propulsion Laboratory, Pasadena. The stand-alone control. package consists 
of an Interdata 70 minicomputer, a Perceptronics +1I/0 Modular Interface, 
and supporting software. It permits generalized computer control of remote 
_ mechanisms in advanced R&D projects. 


Contract No. CS-611109 | 
Price: $24,650 ‘Type: Fixed Price Procurement 
Period of Performance: _ 1] September 1974 -- 11 om 7 1975 


3-04 +1/0 Modular Interface 


Perceptronics has developed a high-speed Modular Interface to connect 
minicomputers to a wide variety of outside processes and devices. The 7 
programmable unit handles analog and digital signals, and provides for both . 
Input and output transfers. Each Interface unit is configured to meet exact 

user needs. Customers for the device have included TRW Systems, U.C.L.A., | 
Purdue University and JPL. Uses have ranged from speech processing to 
manipulator control. — Seo a 4 


Contract No. (Various) | eee ae 
Price: $5,500 (Average) Type: Fixed Price Procurement — 
Period of Performance: 90 day delivery ao 


3-05 Stimulus Programming system | 


| The Southern California Research Institute, Encino, contracted with 

Perceptronics to supply an advanced system for presenting controlled visual 
Stimuli and recording subject responses in studies of industrial pollutant | 
effects. The portable system is built around the Perceptronics +1/0 Modular. — 
Interface, configured to operate in a tape program controlled stand-alone 
mode. See Figures 4-2, 3 and 4-4. | 


Contract No. 75-110b —— | a 
Price: $27,000 Type: Fixed Price Procurement 
Period of Performance: 24 March 1975 -- 1 September 1975 


3-06 Image Enhancement System 


- Perceptronics has been tasked to develop and deliver three systems _ 
for enhancement of real-time video images from an advanced medical 
thermographic scanner. The enhancement system will incorporate a mini-. 
computer, and will perform contrast enhancement and edge enhancement 
routines to improve the diagnostic capability of the scanner. The EQUEneee. | 

is with Paragon Medical Research, Inc. of Encino, California. | 


Contract No. 2014 | | os ae : 
Price: $130,000 Type: Fixed Price Procurement 
Period of Performance: 22 April 1976 - 22 January 1977 


6. STATEMENT OF WORK 


6.1 Genera] 


Perceptronics, Inc. proposes to provide the personnel, facilities, 
materials and services necessary to conduct a 12-month level of effort 


research program directed toward the development of a group decision aiding 
system. | | 


6.2 Program Tasks 


The program will be organized into four phases. The essential 
features of each phase are outlined in the following paragraphs. 


Phase 1 - System Planning Design Specifications 


(1) Select and design group decision test scenario 

(2) Develop group decision aiding algorithms 

(3) Specify and design interactive group input display programs 
(4) Establish overall system software design specifications. 


Phase II - System Prototype Development in Integration 


(1) Design system subprograms 
(2) Code all subprograms and system software 


(3) Integrate all system programs, test and debug on a PDP-11 
system computer 
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Phase III - Experimental Evaluation Studies 
1) Perform evaluation studies and operational system tests 


(2) Design and conduct experimental evaluation studies involving | 
both groups and individual subjects. 


Phase IV - Analysis and Report 


(1) Analyze all experimental data. 
(2) Prepare year-end final report 
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PERCEPTRONICS | 


RESUME - GERSHON WELTMAN 

President 
PROFESSTONAL SPECIALTY 

Human Factors Engineering: — 
PERSONAL ~ 

Born: August 7, 1936 


Married: Two Children 
Security Clearance: Secret 


EDUCATION 
1958 _——B.S. U.C.L.A., Engineering 
1960 M.S. U.C.L.A., Engineering (Biotechnology) 
1962 Ph.D. U.C.L.A., Engineering (Biotechnology) 
1962-3 ~ Fellowship Weizman Institute of Science, Israel 


BACKGROUND AND EXPERIENCE 


Dr. Weltman acts as chief scientist at Perceptronics, Inc. In addition, 
he contributes human factors and bioengineering aspects of system design and — 
development. For the past sixteen years, Dr. Weltman has participated in 
research projects covering a broad area of biotechnical interest, including | 
manual and automatic control, physiological instrumentation, human performance 
in man-machine systems, and human performance underwater (with particular 
reference to the effects of psychological stress). Prior to participating in 
the formation of Perceptronics, Dr. Weltman was a member of the engineering 
faculty at the University of California, Los Angeles. He also served as a 
consultant to industry in the fields of bio-instrumentation for space and 
commercial applications. At Perceptronics, Dr. Weltman has headed a number — 
of research and development projects in the areas of display optimization, . 
man/machine interaction, robotics, and physiological monitoring. He has 

- contributed over 30 papers to scientific and professional journals. 
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| RESUME - AMOS FREEDY 
Vice President 
PROFESSIONAL SPECIALTY 
- Computer Systems, Control Systems and Biotechnology 
PERSONAL 
Born: October 11, 1937 


Married: Two Children 
Security Clearance: Secret 


EDUCATION 
1965 B.S. —ULC.L.A., Engineering | 
1967 M.S. U.C.L.A., Engineering (Control Systems 
and Biotechnology) 7 
1969 Ph.D U.C.L.A., Engineering (Electronics | 
Systems , Control Systems, and Biotechnology). 


BACKGROUND AND EXPERIENCE ~ 


Dr. Freedy is a senior scientist at Perceptronics, Inc., and is re- 

sponsible for system analysis, design and development. For the past six 

years, Dr. Freedy has been actively participating in the broad area of 

computer decisions and control, in the particular the applications of 

~ Jearning and artificial intelligence techniques to man-machine systems. 
Dr. Freedy is the developer of the Autonomous Control Subsystem (ACS) an 
interactive computer learning system that has been used for remote manipu-— 
Tator control and operator decision aiding in continuous command and control. 
Dr. Freedy has also worked in bioengineering where his work included design 
of physiological data acquisition systems, myoelectrical servo controls, and 
development of control systems for artificial arms. Dr. Freedy has also 
served as consultant to industry in the design of physiological instrumenta- 
tion and development of models of biological processes. Dr. Freedy has been 
a principal investigator in numerous projects including computer data — 
acquisition and control. Dr. Freedy has published with his colleagues over 
15 papers in scientific and professional journals. 
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RESUME - SARAH A. S. GOLDBERG 


TITLE 
Director of Administrative Services 
PROFESSIONAL SPECIALTY 
Administrative Management and Clinical Psychology 
"PERSONAL | 
Born: August 26, 1951 


Citizenship: USA 
Security Clearance: Secret 


EDUCATION | 
1973 —s~B..A. U.C.L.A., Psychology | 
1971 Certificate L.A.C.C.,. Psychology 


of Psychological Services 
BACKGROUND AND EXPERIENCE 


As Director of Administrative Services for Perceptronics, Ms. Goldberg's 
principal responsibility is to supervise the administrative functions of the 
office. These include contract administration, personnel management, 
secretarial and support services, purchasing of supplies and equipment, and 
production of all corporate documentation. In addition, Ms. Goldberg acts ~ 
as Security Supervisor. Her duties make her one of the main points of 

“interaction between Perceptronics and the outside world, as well as between 
Perceptronics management and employees. Her professional specialization in — 
clinical psychology often facilitates solution of the problems involved. 
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Brees «ane pei. « 


TITLE 
Consultant 
PROFESSIONAL SPECIALTY | - - 
Decision Analysis, Problem Solving Systems, Artificial Intelligence. | 
PERSONAL | Oo . 


Born: September 4, 1936 
Married: Three Children 


EDUCATION 


— 1960 B.Sc. Technion, Israel, Electronics | oe 
1961 M.Sc. Newark College of Engineering, N.J., Electronics 
1965 M.Sc. Rutgers University, New Brunswick, N.J., Physics 
(1965 Ph.D. Polytechnic Institute of Brooklyn, N.Y., Electronics. 


BACKGROUND AND EXPERIENCE 


Dr. Pearl acts as consultant in decision aiding and problem solving systems 
at Perceptronics, Inc. For the past six years, he has been a Professor in the 
| Engineering Systems Department at UCLA and has done research in the areas of 
decision analysis, problem solving systems, pattern recognition, heuristic - 
learning, inferential question-answering systems, information theory, storage . 
and computational economy, probability assessment, and the theory of - 
--yepresentation. Prior to working at UCLA, Dr. Pearl did pioneering research — 
in superconducting material for RCA in Princeton, N.J. for which he received — 
the RCA Laboratories Achievement Award (1963). He has experience with a 
multitude of professional and academic disciplines. At Perceptronics, Dr. Pearl 
consults in the areas of group decision making, decision tree elicitation, _ 
multi-attribute utility assessment, and probability assessment. | 
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RESUME - ANTONIO LEAL 


TITLE 

Director of Adaptive Systems 
PROFESSIONAL SPECIALTY 

Artificial Intelligence and Computer-Aided Decision Making 
PERSONAL 

Born: December 9, 1942 


Married: Three Children 
Security Clearance: Secret 


EDUCATION 
1965 B.A University of Illinois, Mathematics 
1966 M.S University of Illinois, Mathematics 
1976 Ph.D. University of California at Los es Computer-Aided 


| | Decision Making 
BACKGROUND AND EXPERIENCE 


Dr. Leal acts as Director of Adaptive and Problem Solving Systems at 
Perceptronics, Inc. He supervises projects involving the application of 
heuristic techniques to decision-making and strategy selection in Ballistic 
Missile Defense, the application of planning networks to robot problem solving 
tasks, and the use of decision aiding in anti-submarine warfare. For the 
past six years, Dr. Leal has done research and development at System Development 
Corporation, Santa Monica, California, in the areas of Computer Language Design, 
Robot Problem Solving Systems, Natural Language Processing, Data Base Management 
Systems, Query Language Design, Computer-Aided Decision Making, Man/Machine 

- Interactive Planning Systems, and research into cognitive processes and | 
learning. Prior to working at SDC, Dr. Leal was Project Manager at Computer 
Sciences Corporation in Washington, D.C., where he designed and developed 
Air-Defense Simulation Models and worked on Hospital Information Processing 
Systems. Dr. Leal has experience with a multitude of programming languages 
and software systems. He is currently a member of ACM and MENSA. 
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RESUME - WILLIAM H. CROOKS 
TITLE 
‘Senior Scientist 
PROFESSIONAL SPECIALTY 


Engineering Psychology, Training and Instructional ‘systems, 
Experimental Design and Analysis | 


- PERSONAL - 
Born: January 21, 1944 _ 
Single 
Security Clearance: Secret 

EDUCATION = = 
1966 B.S Oregon State University, Social Science 
1970 M.S University of Illinois, Urbana-Champaign, _ 
; Psychology _ | ~* 

1973 Ph.D. 7 ae University of Illinois, Urbana- -Champaign, © 


Major: Experimental Psychology 
Minor: Computer Science _ 


BACKGROUND AND EXPERIENCE 


Dr. Crooks is a staff scientist at Perceptronics, Inc. His primary 
responsibility is experimental design and analysis for evaluation of human - 
_ performance in advanced man/machine systems. The research topics include | 
- computer-assisted instruction, computer-aided decision making, and display 
system evaluation. Dr. Crooks is a graduate of the Aviation Research 
Laboratory, University of Illinois. While at the Laboratory he conducted 
research in automated adaptive training and developed a number of specialized 
computer programs for experimental control and data analysis. Prior to | 
joining Perceptronics, Dr. Crooks was an applications engineer for a research 
equipment manufacturer. He is an accomplished programmer with several | 
computer languages and is a licensed private pilot. Dr. Crooks is a member 
of the Human Factors Society. | | 


